diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index d9a04f2f03..d88f693627 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -110,6 +110,8 @@ *** xref:manage:kubernetes/k-cluster-property-configuration.adoc[Cluster Properties] *** xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics] *** xref:manage:kubernetes/k-manage-connectors.adoc[Manage Kafka Connect] +*** xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines] +**** xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscaling] *** xref:manage:kubernetes/storage/index.adoc[Storage] **** xref:manage:kubernetes/storage/k-volume-types.adoc[Volume Types] **** xref:manage:kubernetes/storage/k-configure-storage.adoc[Configure Storage] diff --git a/modules/get-started/pages/licensing/overview.adoc b/modules/get-started/pages/licensing/overview.adoc index fe8bfb81ab..8334157879 100644 --- a/modules/get-started/pages/licensing/overview.adoc +++ b/modules/get-started/pages/licensing/overview.adoc @@ -249,6 +249,26 @@ The following enterprise features are available with a valid Enterprise Edition |=== +[[operator]] +=== Redpanda Operator enterprise features + +The Redpanda Operator gates the following Kubernetes features on a valid Enterprise Edition license. In both cases, the operator reads the license from a Secret referenced by the `enterprise.licenseSecretRef` values in the operator Helm chart. + +.Enterprise features in the Redpanda Operator +[cols="1a,2a,2a"] +|=== +| Feature | Description | Restrictions Without Valid License + +| xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[Stretch Clusters] (StretchCluster resource) +| A single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability, managed by the multicluster Redpanda Operator. +| The multicluster operator requires a valid license to start, so stretch clusters cannot be deployed or managed without one. On a running stretch cluster, the operator continuously reports license state through the StretchCluster resource's `LicenseValid` status condition and `status.licenseStatus`, including the expiration date and any enterprise features in violation. When the license expires, the stretch cluster keeps running and its brokers' enterprise features follow the restrictions in <>. + +| xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] (Pipeline resource) +| Run Redpanda Connect pipelines declaratively through the operator's Connect controller. Requires a license that includes the Redpanda Connect product. +| When the license is missing, expired, invalid, or does not include the Redpanda Connect product, the operator stops creating and updating pipeline workloads: new Pipeline resources report `License: False` in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. + +|=== + == Manage licenses for Redpanda Redpanda offers multiple ways to manage your license depending on your deployment method. Proper license management ensures access to advanced enterprise features and avoids restrictions when licenses are invalid or expired. @@ -267,6 +287,24 @@ Redpanda Console offers two methods for applying or updating a license, dependin Redpanda Connect offers multiple ways to apply or update your license. See xref:connect:get-started:licensing.adoc#apply-a-license-key-to-redpanda-connect[Apply a license key to Redpanda Connect]. +== Manage licenses for the Redpanda Operator + +The Redpanda Operator reads its license from a Kubernetes Secret referenced by the `enterprise.licenseSecretRef` values in the operator Helm chart: + +[source,bash] +---- +kubectl create secret generic redpanda-license \ + --from-file=license= \ + --namespace + +helm upgrade --install redpanda-operator redpanda/operator \ + --namespace \ + --set enterprise.licenseSecretRef.name=redpanda-license \ + --set enterprise.licenseSecretRef.key=license +---- + +This operator-level license gates the operator's own enterprise features, such as the <>. The clusters that the operator manages are licensed independently: set the `enterprise.license` or `enterprise.licenseSecretRef` field in a Redpanda or StretchCluster resource, and the operator applies that license to the cluster for you. See xref:get-started:licensing/add-license-redpanda/kubernetes.adoc[]. + == Next steps - xref:get-started:licensing/add-license-redpanda/index.adoc[] diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index 3fd2de832c..a19bb7dcdf 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -22,6 +22,17 @@ The release also adds the xref:reference:rpk/rpk-k8s/rpk-k8s.adoc[`rpk k8s` plug For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[]. +=== Redpanda Connect pipelines (beta) + +The Redpanda Operator introduces the Pipeline custom resource, which runs xref:connect:home:index.adoc[Redpanda Connect] pipelines as declaratively managed Kubernetes workloads. You define the pipeline configuration in the resource, and the operator deploys it, lints it before startup, rolls it automatically on configuration and credential changes, and reports health through status conditions. A pipeline can reference a Redpanda resource and a User resource, and the operator injects broker addresses, TLS material, and SASL credentials into the rendered configuration for you. + +Pipelines also implement the Kubernetes scale subresource, so you can scale them with `kubectl scale`, a HorizontalPodAutoscaler, or KEDA, including on consumer group lag and on the metrics Redpanda Connect emits. + +The Pipeline resource is in beta and requires an Enterprise Edition license. Enable it with the operator chart's `connectController.enabled=true` setting. + +* xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes] -- Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. +* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[Autoscale Redpanda Connect Pipelines] -- Step-by-step HPA and KEDA autoscaling. + === NodePool CRD documentation removed The NodePool custom resource (beta in v26.1.x) is intended for internal use in Bring Your Own Cloud (BYOC) deployments and is no longer documented for self-managed deployments. The CRD remains available and actively maintained, and existing configurations continue to work. diff --git a/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc new file mode 100644 index 0000000000..d146788104 --- /dev/null +++ b/modules/manage/pages/kubernetes/k-autoscale-connect-pipelines.adoc @@ -0,0 +1,314 @@ += Autoscale Redpanda Connect Pipelines +:description: Scale Redpanda Connect pipelines manually or automatically with kubectl scale, a HorizontalPodAutoscaler, or KEDA. +:page-topic-type: how-to +:page-beta: true +:env-kubernetes: true +:page-categories: Management, Integration +:personas: platform_operator, platform_engineer +:learning-objective-1: Scale a pipeline manually through the scale subresource +:learning-objective-2: Autoscale a pipeline on CPU and memory with a HorizontalPodAutoscaler +:learning-objective-3: Autoscale a pipeline on consumer group lag or Connect metrics with KEDA + +// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator. + +Match pipeline capacity to your workload by scaling the number of pipeline replicas, either manually or automatically based on resource usage, consumer group lag, or the metrics that Redpanda Connect emits. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +The xref:manage:kubernetes/k-connect-pipelines.adoc[Pipeline resource] implements the Kubernetes https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[scale subresource^], so anything that speaks the `/scale` API can drive `spec.replicas`: `kubectl scale`, a HorizontalPodAutoscaler (HPA), or a https://keda.sh/[KEDA^] ScaledObject. + +IMPORTANT: Always target the Pipeline resource, never its Deployment. The operator resets the Deployment's replica count to the Pipeline's value on every sync, so an autoscaler pointed at the Deployment is continuously fought and undone. + +== Prerequisites + +You must have the following: + +* A running pipeline managed by the Redpanda Operator with the Connect controller enabled. See xref:manage:kubernetes/k-connect-pipelines.adoc[]. +* For HPA on CPU or memory: https://github.com/kubernetes-sigs/metrics-server[metrics-server^] installed in the cluster. +* For KEDA: https://keda.sh/docs/latest/deploy/[KEDA^] installed in the cluster. +* For scaling on Redpanda Connect metrics: a Prometheus installation that scrapes pipeline Pods, for example https://github.com/prometheus-operator/kube-prometheus[kube-prometheus-stack^] together with the operator chart's `connectController.monitoring.enabled=true` setting. + +Configure only one autoscaler per pipeline. An HPA and a KEDA ScaledObject targeting the same Pipeline fight each other. + +== Scale a pipeline manually + +Scale the Pipeline directly with `kubectl scale`: + +[,bash] +---- +kubectl scale pipeline/ --replicas=3 --namespace +---- + +The operator updates the pipeline's Deployment to match, and the pipeline's consumer group rebalances its partitions across the new replicas. + +To confirm that the scale subresource is wired up, read it directly: + +[,bash] +---- +kubectl get --raw "/apis/cluster.redpanda.com/v1alpha2/namespaces//pipelines//scale" +---- + +The response reports `spec.replicas`, `status.replicas`, and `status.selector`, the label selector that autoscalers use to find the pipeline's Pods. To list a pipeline's Pods yourself, the `app.kubernetes.io/instance=` label is the simplest filter. + +[NOTE] +==== +When you scale a pipeline that consumes from Redpanda, replicas beyond the topic's partition count sit idle, because a consumer group assigns each partition to at most one member. Size `--replicas`, and any autoscaler's maximum, to the partition count of the input topic. +==== + +== Autoscale on CPU and memory with HPA + +CPU-based and memory-based HPAs work without any extra configuration, because pipeline Pods always carry resource requests: if you don't set `spec.resources`, the operator applies default requests of `100m` CPU and `256Mi` memory. + +. Ensure metrics-server is running. If `kubectl top pods` returns metrics, you're ready. Otherwise, install it: ++ +[,bash] +---- +kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml +---- + +. Set explicit resource requests on the pipeline, so utilization percentages are meaningful for your workload: ++ +[,yaml] +---- +spec: + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 300m + memory: 512Mi +---- + +. Create an HPA whose `scaleTargetRef` points at the Pipeline: ++ +.`pipeline-hpa.yaml` +[,yaml] +---- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: orders-to-warehouse + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicas: 1 + maxReplicas: 4 + behavior: + scaleUp: + stabilizationWindowSeconds: 0 + scaleDown: + stabilizationWindowSeconds: 60 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 +---- + +. Apply the manifest and watch the HPA react. Resource metrics lag by about a minute: ++ +[,bash] +---- +kubectl apply -f pipeline-hpa.yaml +kubectl get hpa orders-to-warehouse --namespace --watch +---- ++ +Example output while the pipeline is busy. CPU is at 300% of its request, above the 50% target, so the HPA scales out toward `maxReplicas`: ++ +[.no-copy] +---- +NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE +orders-to-warehouse Pipeline/orders-to-warehouse cpu: 300%/50%, memory: 30%/80% 1 4 4 2m +---- + +. Verify that the Pipeline followed: ++ +[,bash] +---- +kubectl get pipeline orders-to-warehouse --namespace +kubectl get pods -l app.kubernetes.io/instance=orders-to-warehouse --namespace +---- + +To scale an HPA on the metrics Redpanda Connect emits instead of CPU or memory, either expose them to the HPA as custom metrics through https://github.com/kubernetes-sigs/prometheus-adapter[prometheus-adapter^], or use KEDA's Prometheus trigger, described next, which queries Prometheus directly and needs no adapter. + +== Autoscale with KEDA + +KEDA drives the same scale subresource, but adds event-driven triggers, such as consumer group lag and arbitrary Prometheus queries, and can scale a pipeline to zero when it's idle. Under the hood, KEDA materializes and manages an HPA named `keda-hpa-` for you. + +Install KEDA if it isn't already present: + +[,bash] +---- +helm repo add kedacore https://kedacore.github.io/charts +helm upgrade --install keda kedacore/keda --namespace keda --create-namespace +---- + +=== Scale on consumer group lag + +For pipelines that consume from Redpanda, consumer group lag is usually the signal you actually want: it measures how far the pipeline is behind, not how busy its CPUs are. KEDA's `kafka` trigger works with Redpanda because Redpanda is Kafka API-compatible. + +. Create a ScaledObject that targets the Pipeline and watches the pipeline's consumer group: ++ +.`scaledobject-lag.yaml` +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: orders-to-warehouse + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicaCount: 1 + maxReplicaCount: 6 + pollingInterval: 10 # How often KEDA evaluates the trigger (seconds). + cooldownPeriod: 120 # Delay before scaling to zero (only with minReplicaCount: 0). + advanced: + horizontalPodAutoscalerConfig: + behavior: + scaleUp: + stabilizationWindowSeconds: 0 + scaleDown: + stabilizationWindowSeconds: 60 + policies: + - type: Pods + value: 2 + periodSeconds: 30 + triggers: + - type: kafka + metadata: + bootstrapServers: ..svc.cluster.local:9093 + consumerGroup: orders-to-warehouse-ingest + topic: orders + lagThreshold: "500" # Target lag per replica. + offsetResetPolicy: earliest + allowIdleConsumers: "false" # Never scale beyond the partition count. +---- ++ +Replace `..svc.cluster.local:9093` with the bootstrap address of your cluster's internal Kafka listener, and set `consumerGroup` and `topic` to match the pipeline's `input.redpanda` block. ++ +KEDA sizes the pipeline at roughly `total lag / lagThreshold` replicas, clamped between `minReplicaCount` and `maxReplicaCount`. With `allowIdleConsumers: "false"`, KEDA also caps replicas at the topic's partition count, matching consumer group parallelism. + +. If the listener requires TLS or SASL, add a https://keda.sh/docs/latest/scalers/apache-kafka/[TriggerAuthentication^] and reference it from the trigger: ++ +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: TriggerAuthentication +metadata: + name: redpanda-kafka-auth + namespace: +spec: + secretTargetRef: + - parameter: sasl + name: keda-redpanda-auth + key: sasl # For example: scram_sha512 + - parameter: username + name: keda-redpanda-auth + key: username + - parameter: password + name: keda-redpanda-auth + key: password + - parameter: tls + name: keda-redpanda-auth + key: tls # enable + - parameter: ca + name: keda-redpanda-auth + key: ca.crt +---- ++ +Reference it in the ScaledObject trigger with `authenticationRef.name: redpanda-kafka-auth`. The referenced user needs `Describe` permissions on the topic and consumer group to read lag. + +. Apply the manifest and watch the pipeline scale as lag grows and drains: ++ +[,bash] +---- +kubectl apply -f scaledobject-lag.yaml +kubectl get scaledobject orders-to-warehouse --namespace +kubectl get pipeline orders-to-warehouse --namespace --watch +---- ++ +As producers outpace the pipeline, lag crosses the threshold and KEDA raises `spec.replicas`; the consumer group rebalances partitions across the new Pods, lag drains, and after the scale-down stabilization window KEDA scales the pipeline back in. + +=== Scale on Redpanda Connect metrics + +Redpanda Connect emits pipeline-level metrics such as `input_received`, `output_sent`, and latency timings on each Pod's `http` port (4195) at `/metrics`. Scaling on these lets you react to actual pipeline throughput even when the input isn't a Redpanda topic. + +. Get the metrics into Prometheus. With Prometheus Operator installed, set `connectController.monitoring.enabled=true` on the operator chart so it renders a PodMonitor for every pipeline: ++ +[,bash] +---- +helm upgrade --install redpanda-operator redpanda/operator \ + --namespace \ + --reuse-values \ + --set connectController.monitoring.enabled=true +kubectl get podmonitor --namespace +---- + +. Create a ScaledObject with a `prometheus` trigger. This example adds one replica for every 5,000 messages per second the pipeline receives: ++ +.`scaledobject-metrics.yaml` +[,yaml] +---- +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: orders-to-warehouse-throughput + namespace: +spec: + scaleTargetRef: + apiVersion: cluster.redpanda.com/v1alpha2 + kind: Pipeline + name: orders-to-warehouse + minReplicaCount: 1 + maxReplicaCount: 4 + triggers: + - type: prometheus + metadata: + serverAddress: http://prometheus-operated.monitoring.svc:9090 + query: sum(rate(input_received{namespace="", pod=~"orders-to-warehouse-.*"}[2m])) + threshold: "5000" +---- + +. Apply the manifest and confirm that KEDA created its HPA against the Pipeline: ++ +[,bash] +---- +kubectl apply -f scaledobject-metrics.yaml +kubectl get hpa keda-hpa-orders-to-warehouse-throughput --namespace +---- + +=== Scale to zero + +Set `minReplicaCount: 0` on a ScaledObject to let KEDA stop an idle pipeline entirely and restart it when the trigger fires again, for example when lag reappears on the consumer group. KEDA manages the 0 to 1 transition itself, waiting `cooldownPeriod` seconds after the last trigger activity before scaling to zero. A plain HPA cannot do this. + +== How autoscaling interacts with the Pipeline spec + +* *`spec.paused` wins over the autoscaler.* A paused pipeline stays at zero replicas (phase `Stopped`) even while an HPA or KEDA keeps writing `spec.replicas`. When you unpause it, the pipeline resumes at the autoscaler-managed count. +* *An explicit `spec.replicas: 0` is preserved.* The operator does not re-default it to 1. Note that a plain HPA stops managing a target whose replica count is 0; this is standard HPA behavior. Scale back up manually, or use KEDA with `minReplicaCount: 0`, which manages the zero state itself. +* *Stop declaring `replicas` in manifests once an autoscaler owns it.* Every re-apply of a manifest that sets `spec.replicas` overwrites the autoscaler's value. Remove the field from the manifests you apply, or configure your GitOps tool to ignore it, for example with Argo CD's `ignoreDifferences`. + +== Next steps + +* xref:manage:kubernetes/k-connect-pipelines.adoc[] +* https://keda.sh/docs/latest/scalers/[KEDA scalers reference^] +* https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/[Kubernetes HPA documentation^] diff --git a/modules/manage/pages/kubernetes/k-connect-pipelines.adoc b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc new file mode 100644 index 0000000000..64fc7599ca --- /dev/null +++ b/modules/manage/pages/kubernetes/k-connect-pipelines.adoc @@ -0,0 +1,651 @@ += Redpanda Connect Pipelines in Kubernetes +:description: Run Redpanda Connect pipelines declaratively with the Redpanda Operator using the Pipeline resource. +:page-topic-type: how-to +:page-beta: true +:env-kubernetes: true +:page-categories: Management, Integration +:personas: platform_operator, platform_engineer +:learning-objective-1: Enable the Connect controller on the Redpanda Operator +:learning-objective-2: Connect pipelines to Redpanda clusters with scoped credentials +:learning-objective-3: Inject secrets and cloud IAM identities into pipeline workloads + +// DRAFT (DOC-2275): Hold until the Pipeline resource ships in a released operator. +// When rp-connect-docs#457 publishes, add a cross-link to connect:install:kubernetes-operator.adoc in Next steps. + +Run xref:connect:home:index.adoc[Redpanda Connect] pipelines as Kubernetes resources managed by the Redpanda Operator. You define each pipeline in a `Pipeline` custom resource, and the operator deploys it, validates its configuration before startup, rolls it on every change, and reports its health in the resource status. Pipelines follow the same declarative workflow as your Topic and User resources, so you can build and operate streaming data pipelines with the same GitOps tooling you already use for cluster resources. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +This feature is in beta and requires Redpanda Operator v26.2.1 or later. + +include::shared:partial$enterprise-license.adoc[] + +Your license must include the Redpanda Connect product. The operator treats a license without this entitlement like a missing license and does not deploy Pipeline resources. See <>. + +== When to use the Pipeline resource + +Use the Pipeline resource when you want: + +* *Declarative, GitOps-friendly management*: Pipelines are plain Kubernetes manifests that you can version, review, and deploy with tools such as Argo CD or Flux. +* *Validated rollouts*: The operator lints every configuration before the pipeline starts, and rolls the workload automatically when the configuration changes. +* *Cluster integration*: A pipeline can reference a Redpanda resource in the same namespace, and the operator injects the connection details, TLS material, and credentials for you. +* *Built-in connectors*: Redpanda Connect ships hundreds of connectors and processors, including AI integrations, without the JVM or plugin management that Kafka Connect requires. + +For Kafka Connect, which is community-supported with Redpanda, see xref:manage:kubernetes/k-manage-connectors.adoc[]. To run Redpanda Connect on Kubernetes without the operator, see xref:connect:install:helm-chart.adoc[]. + +== Prerequisites + +You must have the following: + +* *Kubectl*: Ensure you have the https://kubernetes.io/docs/tasks/tools/#kubectl[`kubectl`^] command-line tool installed and configured to communicate with your cluster. +* *Redpanda Operator*: Ensure you have the xref:deploy:redpanda/kubernetes/k-production-deployment.adoc[Redpanda Operator] deployed in your Kubernetes cluster. +* *Enterprise Edition license*: The Connect controller is an enterprise feature and requires a license that includes the Redpanda Connect product. Without one, pipelines are reconciled but never deployed. See <>. +* *Image access*: Your cluster must be able to pull the Redpanda Connect image (`docker.redpanda.com/redpandadata/connect` by default). + +== Enable the Connect controller + +The Connect controller ships inside the Redpanda Operator binary and is disabled by default. Enable it through the operator Helm chart, which configures both the controller and the enterprise license: + +. Create a Secret that holds your Enterprise Edition license: ++ +[,bash] +---- +kubectl create secret generic redpanda-license \ + --from-file=license= \ + --namespace +---- + +. Install or upgrade the operator with the Connect controller enabled: ++ +[,bash] +---- +helm upgrade --install redpanda-operator redpanda/operator \ + --namespace \ + --set connectController.enabled=true \ + --set enterprise.licenseSecretRef.name=redpanda-license \ + --set enterprise.licenseSecretRef.key=license +---- ++ +`connectController.enabled: true` adds the `--enable-connect` flag to the operator Deployment, and `enterprise.licenseSecretRef` mounts the license Secret into the operator Pod. To also create a PodMonitor for each pipeline, for example to scrape pipeline metrics with Prometheus Operator, set `connectController.monitoring.enabled=true`. + +. Verify that the Pipeline CRD is installed: ++ +[,bash] +---- +kubectl get crd pipelines.cluster.redpanda.com +---- + +== Deploy your first pipeline + +A pipeline is defined entirely in the `spec.configYaml` field, which holds a standard Redpanda Connect configuration. This minimal example generates data and writes it to the Pod's standard output, so you can try the workflow without a Redpanda cluster: + +. Create a file called `hello-pipeline.yaml`: ++ +.`hello-pipeline.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: hello + namespace: +spec: + configYaml: | + input: + generate: + interval: 1s + mapping: 'root.msg = "hello from connect"' + output: + stdout: {} +---- + +. Apply the manifest and watch the pipeline come up: ++ +[,bash] +---- +kubectl apply -f hello-pipeline.yaml +kubectl get pipeline hello --namespace --watch +---- ++ +Example output: ++ +[.no-copy] +---- +NAME READY PHASE REPLICAS AVAILABLE AGE +hello True Running 1 1 20s +---- + +. Inspect the pipeline's output: ++ +[,bash] +---- +kubectl logs deploy/hello -c connect --namespace -f +---- + +When you apply a Pipeline resource, the operator: + +. Renders a ConfigMap that holds the final `connect.yaml`. +. Renders a Deployment whose `lint` init container runs `redpanda-connect lint` against the configuration. The pipeline only starts if the configuration passes lint. A failing configuration surfaces as `ConfigValid: False` in the resource status. +. Stamps the Pod template with a `cluster.redpanda.com/config-checksum` annotation, so any later configuration change automatically rolls the Deployment. A second annotation, `cluster.redpanda.com/credentials-checksum`, tracks every referenced Secret and ConfigMap, so credential rotations also roll the pipeline. + +To stop a pipeline without deleting it, pause it. The workload scales to zero and the phase changes to `Stopped`: + +[,bash] +---- +kubectl patch pipeline hello --namespace --type merge -p '{"spec":{"paused":true}}' +---- + +== Connect a pipeline to a Redpanda cluster + +Production pipelines read from or write to a Redpanda cluster. Bind the pipeline to a Redpanda resource with `spec.cluster.clusterRef`, and give it SASL credentials with a xref:manage:kubernetes/security/authentication/k-user-controller.adoc[User resource] referenced through `spec.userRef`. The operator resolves the broker addresses, TLS material, and SASL credentials from the referenced resources and injects them into the rendered configuration, so your `configYaml` contains only the pipeline logic. + +This example deploys a pipeline that reads the `orders` topic and archives each order to S3. + +. Create a User resource for the pipeline. The user owns the pipeline's SASL identity, and its ACLs scope what the pipeline can access: ++ +.`orders-to-warehouse-user.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: User +metadata: + name: orders-to-warehouse + namespace: +spec: + cluster: + clusterRef: + name: + authentication: + type: scram-sha-512 + password: + valueFrom: + secretKeyRef: + name: orders-to-warehouse-password + key: password + authorization: + acls: + - type: allow + resource: + type: topic + name: orders + patternType: literal + operations: [Read, Describe] + - type: allow + resource: + type: group + name: orders-to-warehouse-ingest + patternType: literal + operations: [Read, Describe] +---- ++ +[NOTE] +==== +Literal ACLs fit a single, static pipeline like this one. Pipelines whose configuration creates topics (for example, change data capture producing `mysql.*` topics) or consumes with dynamic consumer groups need `patternType: prefixed` grants instead. With a non-superuser pipeline user, a missing group ACL surfaces only as a consumer authorization error at runtime, so scope these grants up front. +==== + +. Create the Pipeline resource, referencing both the cluster and the user: ++ +.`orders-to-warehouse-pipeline.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: orders-to-warehouse + namespace: +spec: + cluster: + clusterRef: + name: # The operator resolves brokers and TLS. + userRef: + name: orders-to-warehouse # The operator resolves the password Secret and SASL mechanism. + valueSources: + - name: S3_SECRET_KEY + source: + secretKeyRef: + name: s3-creds + key: secret_access_key + configYaml: | + input: + redpanda: + # Only the pipeline logic. The operator injects seed_brokers, + # tls, and sasl from clusterRef and userRef. + topics: [orders] + consumer_group: orders-to-warehouse-ingest + pipeline: + processors: + - mapping: | + root = this + root.ingested_at = now() + output: + aws_s3: + bucket: warehouse-orders + region: us-east-2 + credentials: + secret: ${S3_SECRET_KEY} +---- + +. Apply both manifests and verify the pipeline: ++ +[,bash] +---- +kubectl apply -f orders-to-warehouse-user.yaml -f orders-to-warehouse-pipeline.yaml +kubectl get pipeline orders-to-warehouse --namespace +---- + +=== What the operator injects + +The operator merges the resolved connection into every `input.redpanda` and `output.redpanda` block in your configuration, and additionally renders the connection as the top-level `redpanda` block (the shared client), so `redpanda_common` components also work without inline credentials. The rendered configuration for the example above looks like this: + +[,yaml] +---- +input: + redpanda: + seed_brokers: # Injected + - redpanda-0.redpanda..svc.cluster.local.:9093 + - redpanda-1.redpanda..svc.cluster.local.:9093 + - redpanda-2.redpanda..svc.cluster.local.:9093 + tls: # Injected + enabled: true + root_cas_file: /etc/tls/certs/ca/ca.crt + sasl: # Injected + - mechanism: SCRAM-SHA-512 + username: ${REDPANDA_SASL_USERNAME} + password: ${REDPANDA_SASL_PASSWORD} + topics: [orders] # User-supplied + consumer_group: orders-to-warehouse-ingest # User-supplied +# ...plus the same connection rendered as the top-level redpanda block, +# and your pipeline and output sections, unchanged. +---- + +The SASL environment variables resolve from the User resource: the username is the User's name, the mechanism comes from `spec.authentication.type`, and the password is projected from the User's password Secret. + +Keys that you set yourself always win. For example, if you write `seed_brokers` inside `input.redpanda`, the operator leaves your value untouched and injects only the missing keys. This is the escape hatch for pipelines that point one input or output at a different cluster. Blocks for other connectors (`aws_s3`, `mongodb`, and so on) pass through untouched, so new Connect connectors work without operator changes. + +If you omit `userRef`, the operator falls back to the SASL identity of the cluster source (for `clusterRef`, the cluster's bootstrap user). Use a dedicated User with scoped ACLs in production so each pipeline authenticates with least-privilege credentials. + +== Connect to an external cluster + +To target a cluster that isn't managed by a Redpanda resource in the same namespace, such as a Redpanda Cloud cluster, a self-managed deployment outside Kubernetes, or another Kafka-compatible cluster, replace `clusterRef` with `cluster.staticConfiguration` and declare the connection inline. The injection contract is identical to `clusterRef`; only the source of the connection details differs. + +.`cross-region-mirror.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: cross-region-mirror + namespace: +spec: + cluster: + staticConfiguration: + kafka: + brokers: + - kafka.us-east.example.com:9094 + tls: + caCert: # Omit for publicly-issued certificates. + secretKeyRef: + name: external-kafka-ca + key: ca.crt + sasl: + mechanism: SCRAM-SHA-512 # PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512 + username: pipeline-mirror-svc + password: + secretKeyRef: + name: external-kafka-creds + key: password + configYaml: | + input: + redpanda: + topics: [orders] + output: + redpanda: + topic: orders.mirrored +---- + +Note the following: + +* `staticConfiguration` and `userRef` are mutually exclusive: the static configuration carries its own SASL identity, so the API server rejects a Pipeline that sets both. `userRef` also requires `clusterRef`, because without a referenced cluster there is no cluster to authenticate against. +* The CA certificate can come from a Secret (`secretKeyRef`) or a ConfigMap (`configMapKeyRef`). For mutual TLS listeners, additionally set `tls.cert` and `tls.key` to present a client certificate. +* Setting any TLS certificate material enables TLS. `tls: {enabled: true}` alone requests TLS with publicly-issued certificates, and omitting the `tls` block connects without TLS. +* The SASL password also accepts an `inline` value. Avoid inline passwords outside development: the value is stored in plain text in the Pipeline spec. + +== Give a pipeline a cloud IAM identity + +Some connectors call cloud APIs directly, for example to read from Amazon RDS with IAM database authentication, or to write to S3 without static keys. Set `spec.serviceAccountName` to run the pipeline Pod under a dedicated Kubernetes ServiceAccount whose cloud IAM trust is scoped to that one pipeline: IAM Roles for Service Accounts (IRSA) on EKS, Workload Identity on GKE, or Pod Identity on AKS. The operator does not create the ServiceAccount; provision it, with its cloud IAM annotations, alongside your other infrastructure. + +The pipeline's cluster credentials (`cluster` and `userRef`) and its cloud identity (`serviceAccountName`) are independent trust boundaries: either can be set without the other. + +This example runs MySQL change data capture against an Amazon RDS database using IAM authentication, so no database password exists anywhere in the Pipeline, the Pod, or a Secret: + +. Create the ServiceAccount with the IAM role annotation. The IAM role's trust policy must be scoped to this exact ServiceAccount, and its policy must grant `rds-db:connect`: ++ +.`mysql-cdc-orders-sa.yaml` +[,yaml] +---- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: mysql-cdc-orders-rds + namespace: + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam:::role/mysql-cdc-orders-rds +---- + +. Create the Pipeline with `serviceAccountName` set: ++ +.`mysql-cdc-orders.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: mysql-cdc-orders + namespace: +spec: + cluster: + clusterRef: + name: + userRef: + name: mysql-cdc-orders-svc # SASL identity for the Redpanda output + serviceAccountName: mysql-cdc-orders-rds # Cloud IAM boundary for AWS calls + valueSources: + # No MYSQL_PASSWORD: IAM authentication supplies a short-lived token. + - name: MYSQL_HOST + source: + secretKeyRef: + name: mysql-cdc-creds + key: host + - name: MYSQL_USER + source: + secretKeyRef: + name: mysql-cdc-creds + key: username + configYaml: | + input: + mysql_cdc: + # Connect calls rds:GenerateDBAuthToken with the Pod's IAM role and + # uses the 15-minute token as the MySQL password. + dsn: "${MYSQL_USER}@tcp(${MYSQL_HOST}:3306)/shop?tls=skip-verify&allowCleartextPasswords=1" + aws: + enabled: true + region: us-east-2 + endpoint: "${MYSQL_HOST}:3306" + stream_snapshot: true + tables: [orders] + flavor: mysql + output: + redpanda: + topic: mysql.shop.orders + key: '${! @table }' +---- + +The Pod assumes only the annotated role, through the projected ServiceAccount token. `userRef` still gates access to Redpanda, while the ServiceAccount gates access to AWS. + +== Inject values and secrets with valueSources + +`spec.valueSources` projects named values into the pipeline Pod as environment variables, which your configuration references with `${NAME}` interpolation. Each entry is a single, typed pull from one of four sources, and you can mix sources freely in one list: + +* `inline`: A literal value in the Pipeline spec. +* `secretKeyRef`: One key from a Kubernetes Secret. +* `configMapKeyRef`: One key from a ConfigMap. +* `externalSecretRef`: A secret resolved through the https://external-secrets.io/[External Secrets Operator^], backed by a system such as Vault or 1Password. + +Because every value is a named, per-key pull, unused keys in a Secret never leak into the Pod environment, the environment variable name can differ from the Secret's key, and multiple pipelines can pull different keys from the same Secret. + +This example fans one topic out to MongoDB, Snowflake, and MySQL, drawing credentials from all four source types: + +.`orders-fanout.yaml` +[,yaml] +---- +apiVersion: cluster.redpanda.com/v1alpha2 +kind: Pipeline +metadata: + name: orders-fanout + namespace: +spec: + cluster: + clusterRef: + name: + userRef: + name: orders-fanout-svc + valueSources: + # MongoDB Atlas: full connection URI from a Secret. + - name: MONGO_URI + source: + secretKeyRef: + name: mongo-orders-atlas + key: connection_uri + # Snowflake key-pair auth: account and user inline, private key from a + # Secret, passphrase from an External Secrets Operator resource. + - name: SNOWFLAKE_ACCOUNT + source: + inline: "ab12345.us-east-1" + - name: SNOWFLAKE_USER + source: + inline: "PIPELINE_SVC" + - name: SNOWFLAKE_PRIVATE_KEY + source: + secretKeyRef: + name: snowflake-pipeline-svc + key: rsa_key.p8 + - name: SNOWFLAKE_PRIVATE_KEY_PASSPHRASE + source: + externalSecretRef: + name: snowflake-pipeline-svc-passphrase + # MySQL: host from a ConfigMap so the endpoint can rotate without + # touching the Pipeline resource. + - name: MYSQL_USER + source: + inline: "warehouse_writer" + - name: MYSQL_PASSWORD + source: + secretKeyRef: + name: mysql-warehouse-creds + key: password + - name: MYSQL_HOST + source: + configMapKeyRef: + name: warehouse-env + key: mysql_replica_host + configYaml: | + input: + redpanda: + topics: [orders] + consumer_group: orders-fanout + output: + broker: + outputs: + - mongodb: + url: ${MONGO_URI} + database: orders + collection: ingested + operation: insert-one + - snowflake_put: + account: ${SNOWFLAKE_ACCOUNT} + user: ${SNOWFLAKE_USER} + private_key: ${SNOWFLAKE_PRIVATE_KEY} + private_key_pass: ${SNOWFLAKE_PRIVATE_KEY_PASSPHRASE} + database: WAREHOUSE + schema: PUBLIC + stage: "@PIPELINE_STAGE" + - sql_insert: + driver: mysql + dsn: "${MYSQL_USER}:${MYSQL_PASSWORD}@tcp(${MYSQL_HOST}:3306)/warehouse?parseTime=true" + table: orders + columns: [order_id, ingested_at, payload] + args_mapping: | + root = [ this.id, this.ingested_at, this.format_json() ] +---- + +Resolution of every entry is reported by the `ValueSourcesResolved` status condition. + +== Customize the pipeline workload + +The Pipeline spec exposes standard workload settings: + +[,yaml] +---- +spec: + replicas: 3 # Default 1. Autoscalable through the scale subresource. + paused: true # Scale to zero without deleting the resource. + image: docker.redpanda.com/redpandadata/connect: # Override the Connect image. + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + serviceAccountName: my-pipeline-sa # Pod identity. See the cloud IAM section. + nodeSelector: + redpanda.com/node-pool: connect # Pin pipelines to a dedicated node pool. + tolerations: + - key: dedicated + operator: Equal + value: connect + effect: NoSchedule + affinity: {} # Standard Kubernetes affinity, merged with zones. + zones: ["us-east-2a", "us-east-2b"] # Spread Pods across availability zones. + budget: + maxUnavailable: 1 # Creates a PodDisruptionBudget. +---- + +The Connect image is resolved with the following precedence: `spec.image` on the Pipeline, then the operator chart's `connectController.image` values, then the default image baked into the operator binary. + +If a pipeline needs setup work before it starts, such as fetching certificates or waiting for a dependency, add `extraInitContainers`. They run to completion, in order, before the operator's built-in `lint` init container, so anything they stage into a shared volume is visible to both lint and the Connect runtime. Declare the backing volume in `extraVolumes` and mount it into the lint and Connect containers with `extraVolumeMounts`: + +[,yaml] +---- +spec: + extraVolumes: + - name: shared + emptyDir: {} + extraVolumeMounts: # Mounted into the lint and connect containers. + - name: shared + mountPath: /shared + readOnly: true + extraInitContainers: + - name: fetch-certs + image: curlimages/curl:8.11.0 + command: ["sh", "-c", "curl -fsSL $CERT_URL -o /shared/ca.pem"] + volumeMounts: + - name: shared + mountPath: /shared +---- + +The volume names `config`, `cluster-tls-ca`, and `cluster-tls-client` are reserved by the operator. + +== Autoscale pipelines + +The Pipeline resource implements the Kubernetes scale subresource, so `kubectl scale`, a HorizontalPodAutoscaler, or a KEDA ScaledObject can drive the replica count directly. For step-by-step HPA and KEDA examples, see xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. + +== Monitor pipelines + +Redpanda Connect serves its metrics on each pipeline Pod's `http` port (4195) at `/metrics`. If you run Prometheus Operator, set `connectController.monitoring.enabled=true` in the operator chart to render a PodMonitor for every pipeline, so metrics such as `input_received` and `output_sent` are scraped automatically. These metrics can also drive autoscaling. See xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[]. + +== License behavior + +The Connect controller requires a valid Enterprise Edition license that includes the Redpanda Connect product to create or update pipeline workloads: + +* *No license, expired license, or a license without the Redpanda Connect product*: New Pipeline resources are reconciled but get no workload. The resource reports `License: False` in its status conditions and never reaches the `Running` phase. Pipelines that were already running keep processing data, but the operator stops applying spec updates to them until you configure a valid license. +* *License restored*: Reconciliation resumes automatically. You do not need to recreate Pipeline resources. + +The operator mirrors the license into each pipeline's namespace as a Pipeline-owned Secret named `-license`, injected into the Pod as `REDPANDA_LICENSE`, so the Connect runtime's own license checks pass without configuring the license twice. Anyone with permission to read Secrets in a namespace that hosts pipelines can read the license, so scope namespace RBAC accordingly. + +For how licensing works across Redpanda products, see xref:get-started:licensing/overview.adoc[]. + +== Check pipeline status + +[,bash] +---- +kubectl get pipeline --namespace -o yaml +---- + +The `.status.phase` field summarizes the pipeline state: + +[cols="1m,3a"] +|=== +| Phase | Meaning + +| Pending +| The resource is accepted but not yet acted on. + +| Provisioning +| The Deployment is created, but Pods are not yet ready. + +| Running +| The desired number of replicas is ready. + +| Stopped +| The pipeline is paused or scaled to zero. + +| Unknown +| The state could not be determined. +|=== + +The `.status.conditions` array reports each aspect of reconciliation: + +[cols="1m,3a"] +|=== +| Condition | True means + +| Ready +| The pipeline workload is healthy. This is the headline condition. + +| ConfigValid +| The Connect configuration passed lint. + +| ClusterRef +| The cluster source (`clusterRef` or `staticConfiguration`) was resolved. + +| UserRef +| The referenced User's credentials, including its password Secret, were resolved. + +| ValueSourcesResolved +| Every `valueSources` entry resolved to an existing object and key. + +| License +| The operator's enterprise license is valid and includes Redpanda Connect. +|=== + +Two operational guarantees are worth knowing: + +* *Pipeline Pods do not depend on the operator process.* The Deployment is owned by the Pipeline resource, not the operator, so operator restarts and upgrades never interrupt running pipelines. +* *Failures never tear down a running workload.* If a referenced Redpanda resource, User, Secret, ConfigMap, or the license becomes unresolvable, the controller reports the failing condition on the status and stops syncing, but the last-known-good Deployment keeps running. Only deleting the Pipeline resource removes its workload. + +== Troubleshoot + +[cols="1a,2a"] +|=== +| Symptom | Cause and resolution + +| `License: False` +| No valid license, or the license does not include the Redpanda Connect product. Configure `enterprise.licenseSecretRef` in the operator chart. See <>. + +| `ConfigValid: False` +| The `lint` init container rejected the configuration. Run `kubectl logs -c lint` to see the lint error. The condition message carries a truncated copy. + +| `ClusterRef: False` +| The referenced Redpanda resource doesn't exist in the pipeline's namespace, or its CRD isn't installed. A `clusterRef` cannot point to another namespace. + +| Pod stuck in `Init` or `ImagePullBackOff` +| The cluster can't pull the Connect image or an `extraInitContainers` image. Check the image reference and pull secrets. + +| `redpanda_common` errors with `shared client not found` +| The pipeline isn't bound to a cluster. Set `cluster.clusterRef` (with `userRef` for SASL clusters) so the operator renders the top-level `redpanda` block. + +| `Ready: False` with reason `NameConflict` +| Another workload already owns an object with this pipeline's name, such as a ConfigMap or Deployment. Rename the pipeline or remove the conflicting object. The operator refuses to adopt objects it does not own. +|=== + +== Next steps + +* xref:manage:kubernetes/k-autoscale-connect-pipelines.adoc[] +* xref:reference:k-crd-index.adoc[] for the Pipeline CRD source definition +* xref:connect:home:index.adoc[] to explore Redpanda Connect components and configuration +* xref:get-started:licensing/overview.adoc[] diff --git a/modules/manage/pages/kubernetes/k-manage-connectors.adoc b/modules/manage/pages/kubernetes/k-manage-connectors.adoc index 7937fe1d48..2424cd24fc 100644 --- a/modules/manage/pages/kubernetes/k-manage-connectors.adoc +++ b/modules/manage/pages/kubernetes/k-manage-connectors.adoc @@ -4,7 +4,7 @@ :page-categories: Management, Integration :env-kubernetes: true -When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. +When you have Kafka Connect deployed, you can create and manage connectors using Redpanda Console or the Kafka Connect REST API. To build streaming data pipelines with enterprise support and without Kafka Connect infrastructure, consider running xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect pipelines] with the Redpanda Operator instead. include::shared:partial$community-supported-connectors.adoc[] diff --git a/modules/reference/pages/k-crd-index.adoc b/modules/reference/pages/k-crd-index.adoc index cc3f6044cc..9a06401b5c 100644 --- a/modules/reference/pages/k-crd-index.adoc +++ b/modules/reference/pages/k-crd-index.adoc @@ -27,6 +27,10 @@ The CRD source definitions are maintained in the https://github.com/redpanda-dat | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_consoles.yaml[`cluster.redpanda.com_consoles.yaml`^] | Defines Redpanda Console resources. +| Pipeline +| https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_pipelines.yaml[`cluster.redpanda.com_pipelines.yaml`^] +| Defines Redpanda Connect pipeline resources. See xref:manage:kubernetes/k-connect-pipelines.adoc[Redpanda Connect Pipelines in Kubernetes]. + | Schema | https://github.com/redpanda-data/redpanda-operator/blob/main/operator/config/crd/bases/cluster.redpanda.com_schemas.yaml[`cluster.redpanda.com_schemas.yaml`^] | Defines Schema Registry resources.