diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 42d2f4a1e..9500c154b 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -24,6 +24,8 @@ content: start_paths: [docs,'*/docs'] - url: https://github.com/redpanda-data/rp-connect-docs branches: main + - url: https://github.com/redpanda-data/adp-docs + branches: main ui: bundle: url: https://github.com/redpanda-data/docs-ui/releases/latest/download/ui-bundle.zip diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 6c63320d7..5af2b8c69 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -23,17 +23,6 @@ *** xref:get-started:cluster-types/byoc/remote-read-replicas.adoc[] ** xref:get-started:cluster-types/create-dedicated-cloud-cluster.adoc[] -* xref:ai-agents:index.adoc[Agentic AI] -** xref:ai-agents:adp-overview.adoc[ADP Overview] -** xref:ai-agents:mcp/index.adoc[MCP] -*** xref:ai-agents:mcp/overview.adoc[Overview] -*** xref:ai-agents:mcp/quickstart.adoc[Quickstart] -*** xref:ai-agents:mcp/configuration.adoc[Configure] -** xref:ai-agents:observability/index.adoc[Transcripts] -*** xref:ai-agents:observability/concepts.adoc[Concepts] -** xref:ai-agents:ai-gateway/index.adoc[AI Gateway] -*** xref:ai-agents:ai-gateway/what-is-ai-gateway.adoc[Overview] - * xref:develop:connect/about.adoc[Redpanda Connect] ** xref:develop:connect/connect-quickstart.adoc[Quickstart] ** xref:develop:connect/configuration/about.adoc[] diff --git a/modules/ai-agents/images/agent-exit-conditions.png b/modules/ai-agents/images/agent-exit-conditions.png deleted file mode 100644 index 5d14a08a4..000000000 Binary files a/modules/ai-agents/images/agent-exit-conditions.png and /dev/null differ diff --git a/modules/ai-agents/images/agent-reasoning-loop.png b/modules/ai-agents/images/agent-reasoning-loop.png deleted file mode 100644 index 2cf208177..000000000 Binary files a/modules/ai-agents/images/agent-reasoning-loop.png and /dev/null differ diff --git a/modules/ai-agents/pages/adp-overview.adoc b/modules/ai-agents/pages/adp-overview.adoc deleted file mode 100644 index 408bcf6cb..000000000 --- a/modules/ai-agents/pages/adp-overview.adoc +++ /dev/null @@ -1,86 +0,0 @@ -= Redpanda Agentic Data Plane Overview -:description: Enterprise-grade infrastructure for building, deploying, and governing AI agents at scale with compliance-grade audit trails. -:page-topic-type: overview -:page-categories: AI, Agentic AI -:personas: evaluator, ai_agent_developer, platform_admin -:learning-objective-1: Identify the key components of Redpanda ADP and their purposes -:learning-objective-2: Describe how each component addresses enterprise governance and reliability requirements -:learning-objective-3: Determine whether Redpanda ADP fits your organization's requirements for AI agent deployment - -include::ai-agents:partial$adp-la.adoc[] - -As glossterm:AI agent[,AI agents] evolve from experimental prototypes to business-critical systems, companies face new challenges. How do you ensure your AI agents are reliable? How do you maintain control over costs and compliance? And how do you scale them across your organization without creating technical debt? - -Teams across your organization want AI agents in production with direct access to enterprise data, from real-time event streams to databases and business systems. Building an agent is the easy part. Running one safely at scale remains the challenge: every database, queue, and API needs its own access policies, creating security gaps and slowing deployment. When you manage high-volume, event-driven data, you need a centralized layer through which all agent interactions flow so that agents can contextualize and act on that data in real time without compromising governance. - -Redpanda Agentic Data Plane (ADP) solves these problems by bringing together key capabilities: a solid data foundation, over 300 proven connectors, and a declarative approach to building AI agents. The result is a unified platform that automatically tracks every agent decision for compliance and audit requirements. - -After reading this page, you will be able to: - -* [ ] {learning-objective-1} -* [ ] {learning-objective-2} -* [ ] {learning-objective-3} - -== AI agents - -With Redpanda AI agents, you declare the agent behavior you want and Redpanda handles execution and orchestration. Instead of writing Python or JavaScript, you define behaviors in YAML. You can orchestrate multiple specialized glossterm:subagent[,sub-agents], or bring your own frameworks like LangChain or LlamaIndex. - -What makes this practical at scale is xref:develop:connect/about.adoc[Redpanda Connect]. More than 300 connectors with built-in filtering, enrichment, and routing give declarative definitions real power. Upcoming templates will provide default behaviors for common domains such as customer success, legal, and finance. - -The result is faster time-to-production, lower maintenance (declarative definitions instead of imperative code), and organizational consistency across teams. - -== MCP servers - -glossterm:MCP server[,MCP servers] translate agent intent into connections to databases, queues, HRIS, CRMs, and other business systems. They are the simplest way to give agents context and capabilities without writing glue code. - -Under the hood, MCP servers wrap the same proven connectors that power some of the world's largest e-commerce, EV, electricity, and AI companies. MCP servers are lightweight, support OIDC-based authentication, and enforce deterministic policies at the tool level. You define tools in YAML, and policy enforcement programmatically prevents prompt injection, SQL injection, and other agent-based attacks. - -With over 300 connectors and real-time debugging capabilities, you reduce integration time while getting enterprise-grade security. You can reuse your existing infrastructure and data sources rather than building new integrations from scratch. - -For more information, see xref:ai-agents:mcp/overview.adoc[MCP Servers Overview]. - -== Transcripts - -Every agent action is recorded in an end-to-end execution log. A single glossterm:transcript[] can span multiple agents, tools, and models, covering interactions that last minutes to days. - -Transcripts are the keystone of agent governance. They are built on Redpanda's immutable log with transcript consensus and TLA+ correctness proofs. No gaps, no tampering. For regulated industries that require multi-year audit trails, this provides a compliance-grade record of every decision an agent makes and every data source it uses. - -Redpanda captures 100% of agent actions through OpenTelemetry standards, with end-to-end lineage across the entire execution chain. You can materialize execution logs to Iceberg tables for long-term retention and analysis, or replay them to evaluate and improve agent performance over time. - -For more information, see xref:ai-agents:observability/concepts.adoc[Transcripts Overview]. - -== AI Gateway - -The AI Gateway manages LLM provider access with two priorities: keeping your application up and keeping costs under control. - -For high availability, the gateway provides provider-agnostic routing with intelligent failover. Your users don't care which provider serves a request. They care that the application stays up. For fiscal control, you get per-tenant budgets and rate limiting, so there are no runaway costs and no surprise bills. - -The gateway also supports tenancy modeling for teams, individuals, applications, and service accounts, giving you chargeback transparency for internal cost allocation. You can proxy both models and MCP gateways, centralizing compliance for all LLM interactions without locking into any single provider. - -For more information, see xref:ai-agents:ai-gateway/what-is-ai-gateway.adoc[AI Gateway Overview]. - -== Enterprise governance - -Redpanda ADP addresses critical enterprise requirements across all components. - -* *Security by design*: MCP servers enforce policies at the tool level, programmatically preventing prompt injection, SQL injection, and other agent-based attacks. Policy enforcement is deterministic and controlled. Agents cannot bypass security constraints even through creative prompting. - -* *Unified authorization*: All components use OIDC-based authentication with an on-behalf-of authorization model. When a user invokes an agent, the agent inherits the intersection of its own permissions and the user's permissions. This ensures proper data access scoping. - -* *Complete observability*: Redpanda ADP provides two levels of inspection. Execution logs (transcripts) capture every agent action with 100% sampling using OpenTelemetry standards. Real-time debugging tools allow you to inspect individual MCP server calls down to individual tool invocations with full timing data. You can view detailed agent actions in glossterm:Redpanda Console[] and replay data for agent evaluations. - -* *Compliance and audit*: For industries requiring multi-year audit trails, Redpanda ADP records every agent action and data source used in decision-making. Execution logs are stored in Redpanda topics and can be materialized to Iceberg tables for long-term retention and analysis. - -== Use cases - -Some ways organizations can leverage Redpanda ADP include: - -* *Automate operational workflows*: Create specialized agents for building management, infrastructure monitoring, compliance reporting, and other domain-specific tasks. -* *Monitor manufacturing and operations*: Deploy multi-agent systems that analyze factory machine telemetry in real-time, detect anomalies, search equipment manuals, and create maintenance tickets automatically. -* *Extend enterprise productivity tools*: Integrate Microsoft Copilot or other workplace agents with internal data sources and systems that are otherwise inaccessible. - -== Next steps - -* xref:ai-agents:mcp/overview.adoc[MCP Server Overview] -* xref:ai-agents:observability/concepts.adoc[Transcripts Overview] -* xref:ai-agents:ai-gateway/what-is-ai-gateway.adoc[AI Gateway Overview] diff --git a/modules/ai-agents/pages/ai-gateway/index.adoc b/modules/ai-agents/pages/ai-gateway/index.adoc deleted file mode 100644 index 3bbcbd3ba..000000000 --- a/modules/ai-agents/pages/ai-gateway/index.adoc +++ /dev/null @@ -1,6 +0,0 @@ -= AI Gateway -:description: Keep AI-powered apps running with automatic provider failover, prevent runaway spend with centralized budget controls, and govern access across teams, apps, and service accounts. -:page-layout: index -:personas: platform_admin, app_developer, evaluator - -include::ai-agents:partial$adp-la.adoc[] \ No newline at end of file diff --git a/modules/ai-agents/pages/ai-gateway/what-is-ai-gateway.adoc b/modules/ai-agents/pages/ai-gateway/what-is-ai-gateway.adoc deleted file mode 100644 index 1f78d4858..000000000 --- a/modules/ai-agents/pages/ai-gateway/what-is-ai-gateway.adoc +++ /dev/null @@ -1,170 +0,0 @@ -= What is an AI Gateway? -:description: Understand how AI Gateway keeps AI-powered apps highly available across providers and prevents runaway AI spend with centralized cost governance. -:page-topic-type: concept -:personas: evaluator, app_developer, platform_admin -:learning-objective-1: Explain how AI Gateway keeps AI-powered apps highly available through governed provider failover -:learning-objective-2: Describe how AI Gateway prevents runaway AI spend with centralized budget controls and tenancy-based governance -:learning-objective-3: Identify when AI Gateway fits your use case based on availability requirements, cost governance needs, and multi-provider or MCP tool usage - -include::ai-agents:partial$adp-la.adoc[] - -Redpanda AI Gateway keeps your AI-powered applications highly available and your AI spend under control. It sits between your applications and the LLM providers and AI tools they depend on. If a provider goes down, the gateway provides automatic failover to keep your apps running. It also offers centralized budget controls to prevent runaway costs. For platform teams, it adds governance at the model-fallback level, tenancy modeling for teams, individuals, apps, and service accounts, and a single proxy layer for both LLM models and glossterm:MCP server[,MCP servers]. - -== The problem - -Modern AI applications face two business-critical challenges: staying up and staying on budget. - -First, applications typically hardcode provider-specific SDKs. An application using OpenAI's SDK cannot easily switch to Anthropic or Google without code changes and redeployment. When a provider hits rate limits, suffers an outage, or degrades in performance, your application goes down with it. Your end users don't care which provider you use; they care that the app works. - -Second, costs can spiral without centralized controls. Without a single view of token consumption across teams and applications, it's difficult to attribute costs to specific customers, features, or environments. Testing and debugging can generate unexpected bills, and there's no way to enforce budgets or rate limits per team, application, or service account. The result: runaway spend that finance discovers only after the fact. - -These two challenges are compounded by fragmented observability across provider dashboards, which makes it harder to detect availability issues or cost anomalies in time to act. And as organizations adopt glossterm:AI agent[,AI agents] that call glossterm:MCP tool[,MCP tools], the lack of centralized tool governance adds another dimension of uncontrolled cost and risk. - -== What AI Gateway solves - -Redpanda AI Gateway delivers two core business outcomes, high availability and cost governance, backed by platform-level controls that set it apart from simple proxy layers. - -=== High availability through governed failover - -Your end users don't care whether you use OpenAI, Anthropic, or Google: they care that your app stays up. AI Gateway lets you configure provider pools with automatic failover, so when your primary provider hits rate limits, times out, or returns errors, the gateway routes requests to a fallback provider with no code changes and no downtime for your users. - -Unlike simple retry logic, AI Gateway provides governance at the failover level: you define which providers fail over to which, under what conditions, and with what priority. This controlled failover can significantly improve uptime even during extended provider outages. - -=== Cost governance and budget controls - -AI Gateway gives you centralized fiscal control over AI spend. Set monthly budget caps for each gateway, enforce them automatically, and set rate limits per team, environment, or application. No more runaway costs discovered after the fact. - -You can route requests to different models based on user attributes. For example, to direct premium users to a more capable model while routing free tier users to a cost-effective option, use a CEL expression. For example: - -[source,cel] ----- -// Route premium users to best model, free users to cost-effective model -request.headers["x-user-tier"] == "premium" - ? "anthropic/claude-opus-4.6" - : "anthropic/claude-sonnet-4.5" ----- - -You can also set different rate limits and spend limits for each environment to prevent staging or development traffic from consuming production budgets. - -=== Tenancy and access governance - -AI Gateway provides multi-tenant isolation by design. Create separate gateways for teams, individual developers, applications, or service accounts, each with their own budgets, rate limits, routing policies, and observability scope. This tenancy model lets platform teams govern who uses what, how much they spend, and which models and tools they can access, without building custom authorization layers. - -=== Unified LLM access (single endpoint for all providers) - -AI Gateway provides a single OpenAI-compatible endpoint that routes requests to multiple LLM providers. Instead of integrating with each provider's SDK separately, you configure your application once and switch providers by changing only the model parameter. - -Without AI Gateway, you need different SDKs and patterns for each provider: - -[source,python] ----- -# OpenAI -from openai import OpenAI -client = OpenAI(api_key="sk-...") -response = client.chat.completions.create( - model="gpt-5.2", - messages=[{"role": "user", "content": "Hello"}] -) - -# Anthropic (different SDK, different patterns) -from anthropic import Anthropic -client = Anthropic(api_key="sk-ant-...") -response = client.messages.create( - model="claude-sonnet-4.5", - max_tokens=1024, - messages=[{"role": "user", "content": "Hello"}] -) ----- - -With AI Gateway, you use the OpenAI SDK for all providers: - -[source,python] ----- -from openai import OpenAI - -# Single configuration, multiple providers -client = OpenAI( - base_url="", - api_key="your-redpanda-token", -) - -# Route to OpenAI -response = client.chat.completions.create( - model="openai/gpt-5.2", - messages=[{"role": "user", "content": "Hello"}] -) - -# Route to Anthropic (same code, different model string) -response = client.chat.completions.create( - model="anthropic/claude-sonnet-4.5", - messages=[{"role": "user", "content": "Hello"}] -) - -# Route to Google Gemini (same code, different model string) -response = client.chat.completions.create( - model="google/gemini-2.0-flash", - messages=[{"role": "user", "content": "Hello"}] -) ----- - -To switch providers, you change only the `model` parameter from `openai/gpt-5.2` to `anthropic/claude-sonnet-4.5`. No code changes or redeployment needed. - -=== Proxy for LLM models and MCP servers - -AI Gateway acts as a single proxy layer for both LLM model requests and MCP servers. For LLM traffic, it provides a unified endpoint. For AI agents that use MCP tools, it aggregates multiple MCP servers and provides deferred tool loading, which dramatically reduces token costs. - -Without AI Gateway, agents typically load all available MCP tools from multiple MCP servers at startup. This approach sends 50+ tool definitions with every request, creating high token costs (thousands of tokens per request), slow agent startup times, and no centralized governance over which tools agents can access. - -With AI Gateway, you configure approved MCP servers once, and the gateway loads only search and orchestrator tools initially. Agents query for specific tools only when needed, which often reduces token usage by 80-90% depending on your configuration and the number of tools aggregated. You also gain centralized approval and governance over which MCP servers your agents can access. - -For complex workflows, AI Gateway provides a JavaScript-based orchestrator tool that reduces multi-step workflows from multiple round trips to a single call. For example, you can create a workflow that searches a vector database and, if the results are insufficient, falls back to web search—all in one orchestration step. - -=== Unified observability and cost tracking - -AI Gateway provides a single dashboard that tracks all LLM traffic across providers, eliminating the need to switch between multiple provider dashboards. - -The dashboard tracks request volume for each gateway, model, and provider, along with token usage for both prompt and completion tokens. You can view estimated spend per model with cross-provider comparisons, latency metrics (p50, p95, p99), and errors broken down by type, provider, and model. - -This unified view helps you answer critical questions such as which model is the most cost-effective for your use case, why a specific user request failed, how much your staging environment costs each week, and what the latency difference is between providers for your workload. - -ifdef::ai-hub-available[] -== Gateway modes - -AI Gateway supports two modes to accommodate different organizational needs: - -*AI Hub Mode* provides zero-configuration access with pre-configured backend pools and intelligent routing. Platform admins simply add provider credentials (OpenAI, Anthropic, Google Gemini), and all teams immediately benefit from 17 routing rules and 6 backend pools. Users can toggle preferences like vision routing or long-context routing, but the underlying architecture is managed by Redpanda. This mode eliminates the complexity of LLM gateway configuration. IT adds API keys once, and all teams benefit immediately. - -*Custom Mode* provides full control over routing rules, backend pools, rate limits, and policies. Admins configure every aspect of the gateway to meet specific requirements. This mode is ideal when you need custom routing logic based business rules, specific failover behavior, or integration with custom infrastructure like Azure OpenAI or AWS Bedrock. - -To understand which mode fits your use case, see xref:ai-gateway/gateway-modes.adoc[]. -endif::[] - -== Common gateway patterns - -Some common patterns for configuring gateways include: - -* *Team isolation*: When multiple teams share infrastructure but need separate budgets and policies, create one gateway for each team. For example, you might configure Team A's gateway with a $5K/month budget for both staging and production environments, while Team B's gateway has a $10K/month budget with different rate limits. Each team sees only their own traffic in the observability dashboards, providing clear cost attribution and isolation. -* *Environment separation*: To prevent staging traffic from affecting production metrics, create separate gateways for each environment. Configure the staging gateway with lower rate limits, restricted model access, and aggressive cost controls to prevent runaway expenses. The production gateway can have higher rate limits, access to all models, and alerting configured to detect anomalies. -* *Primary and fallback for reliability*: To ensure uptime during provider outages, configure provider pools with automatic failover. For example, you can set OpenAI as your primary provider (preferred for quality) and configure Anthropic as the fallback that activates when the gateway detects rate limits or timeouts from OpenAI. Monitor the fallback rate to detect primary provider issues early, before they impact your users. -* *A/B testing models*: To compare model quality and cost without dual integration, route a percentage of traffic to different models. For example, you can send 80% of traffic to `claude-sonnet-4.5` and 20% to `claude-opus-4.6`, then compare quality metrics and costs in the observability dashboard before adjusting the split. -* *Customer-based routing*: For SaaS products with tiered pricing (for example, free, pro, enterprise), use CEL routing based on request headers to match users with appropriate models. - -== When to use AI Gateway - -AI Gateway is ideal for organizations that: - -* Use or plan to use multiple LLM providers -* Need centralized cost tracking and budgeting -* Want to experiment with different models without code changes -* Require high availability during provider outages -* Have multiple teams or customers using AI services -* Build AI agents that need MCP tool aggregation -* Need unified observability across all AI traffic - -AI Gateway may not be necessary if: - -* You only use a single provider with simple requirements -* You have minimal AI traffic (< 1000 requests/day) -* You don't need cost tracking or policy enforcement -* Your application doesn't require provider switching - diff --git a/modules/ai-agents/pages/index.adoc b/modules/ai-agents/pages/index.adoc deleted file mode 100644 index 2b8d0cdd4..000000000 --- a/modules/ai-agents/pages/index.adoc +++ /dev/null @@ -1,6 +0,0 @@ -= Redpanda Agentic Data Plane -:description: Redpanda Agentic Data Plane (ADP) provides enterprise-grade infrastructure for building, deploying, and governing AI agents at scale with enterprise governance, cost controls, and compliance-grade audit trails. -:page-layout: index -:page-aliases: develop:agents/about.adoc, develop:ai-agents/about.adoc - -include::ai-agents:partial$adp-la.adoc[] \ No newline at end of file diff --git a/modules/ai-agents/pages/mcp/configuration.adoc b/modules/ai-agents/pages/mcp/configuration.adoc deleted file mode 100644 index 3a9a08def..000000000 --- a/modules/ai-agents/pages/mcp/configuration.adoc +++ /dev/null @@ -1,211 +0,0 @@ -= Configure the Redpanda Cloud Management MCP Server -:page-aliases: mcp/local/configuration.adoc -:page-beta: true -:description: Learn how to configure the Redpanda Cloud Management MCP Server, including auto and manual client setup, enabling deletes, and security considerations. -:page-topic-type: how-to -:personas: agent_developer, platform_admin -// Reader journey: "I customize and configure" -// Learning objectives - what readers can learn from this page: -:learning-objective-1: Configure MCP clients -:learning-objective-2: Enable delete operations safely -:learning-objective-3: Troubleshoot common configuration issues - -After installing the Redpanda Cloud Management MCP Server, you can configure it for different AI clients, customize security settings, and troubleshoot common issues. - -After reading this page, you will be able to: - -* [ ] {learning-objective-1} -* [ ] {learning-objective-2} -* [ ] {learning-objective-3} - -== Prerequisites - -* At least version 25.2.3 of xref:manage:rpk/rpk-install.adoc[`rpk` installed on your local machine] -* Access to a Redpanda Cloud account -* An MCP-compatible AI client such as Claude, Claude Code, or another tool that supports MCP - -TIP: The MCP server exposes Redpanda Cloud API endpoints for both the link:https://docs.redpanda.com/api/doc/cloud-controlplane/[Control Plane] and the link:https://docs.redpanda.com/api/doc/cloud-dataplane/[Data Plane]. Available endpoints depend on your `rpk` version. Keep `rpk` updated to access new Redpanda Cloud features through the MCP server. New MCP endpoints are documented in Redpanda link:https://github.com/redpanda-data/redpanda/releases[release notes]. - -== Install the integration for Claude or Claude Code - -For some supported clients, you can install and configure the MCP integration using the `rpk cloud mcp install` command. -For Claude and Claude Code, run one of these commands: - -[source,bash] ----- -# Choose one -rpk cloud mcp install --client claude -rpk cloud mcp install --client claude-code ----- - -If you need to update the integration, re-run the install command for your client. - -== Configure other MCP clients manually - -If you're using another MCP-compatible client, manually configure it to use the Redpanda Cloud Management MCP Server. Follow these steps: - -Add an MCP server entry to your client's configuration (example shown in JSON). Adjust paths for your system. - -[,json] ----- -"mcpServers": { - "redpandaCloud": { - "command": "rpk", - "args": [ - "--config", - "", <1> - "cloud", - "mcp", - "stdio" - ] - } -} ----- -<1> Optional: The `--config` flag lets you target a specific `rpk.yaml`, which contains the configuration for connecting to your cluster. Always use the same configuration path as you used for `rpk cloud login` to ensure it has your token. Default paths vary by operating system. See the xref:reference:rpk/rpk-cloud/rpk-cloud-login.adoc[`rpk cloud login`] reference for the default paths. - -You can also <>. - - -== Enable delete operations - -The server disables destructive operations by default. To allow delete operations, add `--allow-delete` to the MCP server invocation. - -CAUTION: Enabling delete operations permits actions like **deleting topics or clusters**. Restrict access to your AI client and double-check prompts. - -[tabs] -==== -Auto-configured clients:: -+ --- -[,bash] ----- -# Choose one -rpk cloud mcp install --client claude --allow-delete -rpk cloud mcp install --client claude-code --allow-delete ----- --- -Manual configuration example:: -+ --- -[,json] ----- -"mcpServers": { - "redpandaCloud": { - "command": "rpk", - "args": [ - "cloud", - "mcp", - "stdio", - "--allow-delete" - ] - } -} ----- --- -==== - -== Specify configuration file paths - -All `rpk` commands accept a `--config` flag, which lets you specify the exact `rpk.yaml` configuration file to use for connecting to your Redpanda cluster. This flag overrides the default search path and ensures that the command uses the credentials and settings from the file you provide. - -Always use the same configuration path for both `rpk cloud login` and any MCP server setup or install commands to avoid authentication issues. By default, `rpk` searches for config files in standard locations depending on your operating system. See the xref:reference:rpk/rpk-cloud/rpk-cloud-login.adoc[reference documentation] for details. Use an absolute path and make sure your user has read and write permissions. - -CAUTION: The `rpk` configuration file contains your Redpanda Cloud token. Keep the file secure and never share it. - -For example, if you want to use a custom config path, specify it for both login and the MCP install command: - -[source,bash] ----- -rpk cloud login --config /Users//my-rpk-config.yaml -rpk cloud mcp install --client claude --config /Users//my-rpk-config.yaml ----- - -Or for Claude Code: - -[source,bash] ----- -rpk cloud login --config /Users//my-rpk-config.yaml -rpk cloud mcp install --client claude-code --config /Users//my-rpk-config.yaml ----- - -== Remove the MCP server - -To remove the MCP server, delete or disable the `mcpServers.redpandaCloud` entry in your client's config (steps vary by client). - -== Security considerations - -* Avoid enabling `--allow-delete` unless required. -* For most local use cases, such as with Claude or Claude Code, log in with your personal Redpanda Cloud user account for better security and easier management. -* If you are deploying the MCP server as part of an application or shared environment, consider using a xref:security:cloud-authentication.adoc#authenticate-to-the-cloud-api[service account] with tailored roles. To log in as a service account, use: -+ -[source,bash] ----- -rpk cloud login --client-id --client-secret --save ----- -* Regularly review and rotate your credentials. - -== Troubleshooting - - -=== Verify your installation - -. Make sure you are using at least version 25.2.3 of `rpk`. -. If you see authentication errors, run `rpk cloud login` again. -. Ensure you installed for the right client: -+ -[source,bash] ----- -rpk cloud mcp install --client claude -# or -rpk cloud mcp install --client claude-code ----- -. If using another MCP client, verify your `mcpServers.redpandaCloud` entry (paths, JSON syntax, and args order). -+ -[[local]] -. Start the server manually using the `rpk cloud mcp stdio` command (one-time login required) to verify connectivity to Redpanda Cloud endpoints: - -+ -[,bash] ----- -rpk cloud login -rpk cloud mcp stdio ----- -+ -.. Send the following newline-delimited JSON-RPC messages (each on its own line): -+ -[source,json] ----- -{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{"roots":{},"sampling":{},"elicitation":{}},"clientInfo":{"name":"ManualTest","version":"0.1.0"}}} -{"jsonrpc":"2.0","method":"notifications/initialized"} -{"jsonrpc":"2.0","id":2,"method":"tools/list"} ----- -+ -Expected response shapes (examples): -+ -[source,json] ----- -{"jsonrpc":"2.0","id":1,"result":{"capabilities":{...}}} -{"jsonrpc":"2.0","id":2,"result":{"tools":[{"name":"...","description":"..."}, ...]}} ----- -+ -.. Stop the server with `Ctrl+C`. - -=== Client can't find the MCP server - -* Re-run the install for your MCP client. -* Confirm the path in `--config /path/to/rpk.yaml` exists and is readable. -* Double-check your client's configuration format and syntax. - -=== Unauthorized errors or token errors - -Your capabilities depend on your Redpanda Cloud account permissions. If an operation fails with a permissions error, contact your account admin. - -* Run `rpk cloud login` to refresh the token. -* Ensure your account has the necessary permissions for the requested operation. - -=== Deletes not working - -* The server disables delete operations by default. Add `--allow-delete` to the server invocation (auto or manual configuration) and restart the client. -* For auto-configured clients, you may need to edit the generated config or re-run the install command and adjust the entry. - - diff --git a/modules/ai-agents/pages/mcp/index.adoc b/modules/ai-agents/pages/mcp/index.adoc deleted file mode 100644 index 4528d3a10..000000000 --- a/modules/ai-agents/pages/mcp/index.adoc +++ /dev/null @@ -1,7 +0,0 @@ -= Model Context Protocol (MCP) -:description: Give AI agents direct access to your databases, queues, CRMs, and other business systems without writing custom glue code. -:page-layout: index - -AI agents need context from your business systems. The Model Context Protocol (MCP) translates agent intent into real connections to databases, queues, CRMs, HRIS, and other systems of record, without you writing custom integration code. - -The Redpanda Cloud Management MCP Server connects your local AI development environment to manage Redpanda Cloud resources. diff --git a/modules/ai-agents/pages/mcp/overview.adoc b/modules/ai-agents/pages/mcp/overview.adoc deleted file mode 100644 index 84aaddd8d..000000000 --- a/modules/ai-agents/pages/mcp/overview.adoc +++ /dev/null @@ -1,73 +0,0 @@ -= Redpanda Cloud Management MCP Server -:page-aliases: mcp/local/overview.adoc, mcp/local/index.adoc -:page-beta: true -:description: Let AI agents securely operate your Redpanda Cloud clusters, topics, and users through natural language commands. -:page-topic-type: overview -:personas: evaluator, agent_developer, platform_admin -// Reader journey: "I'm new" -// Learning objectives - what readers should understand after reading this page: -:learning-objective-1: Explain what the Redpanda Cloud Management MCP Server does -:learning-objective-2: Identify what operations are available through MCP -:learning-objective-3: Identify security considerations for MCP authentication - -The Redpanda Cloud Management MCP Server lets AI agents securely access and operate your Redpanda Cloud account and clusters through natural language commands. - -After reading this page, you will be able to: - -* [ ] {learning-objective-1} -* [ ] {learning-objective-2} -* [ ] {learning-objective-3} - -image::shared:cloud-mcp.gif[A terminal window showing Claude Code invoking the Redpanda Cloud Management MCP Server to list topics in a cluster.] - -== What you can do - -MCP provides controlled access to: - -* link:https://docs.redpanda.com/api/doc/cloud-controlplane/[Control Plane] APIs, such as creating a Redpanda Cloud cluster or listing clusters. -* link:https://docs.redpanda.com/api/doc/cloud-dataplane/[Data Plane] APIs, such as creating topics or listing topics. - -The MCP server runs on your computer and authenticates to Redpanda Cloud using a Redpanda Cloud token. - -You can do anything that's available in the Control Plane or Data Plane APIs. Typical requests you can make to your assistant once connected include: - -* Create a Redpanda Cloud cluster named `dev-mcp`. -* List topics in `dev-mcp`. -* Create a topic `orders-raw` with 6 partitions. - -NOTE: The MCP server does **not** expose delete endpoints by default. You can enable delete endpoints when you create the server if you intentionally want to allow delete operations. - -== Use cases - -* Test automation: Create short-lived clusters, create topics, and validate pipelines quickly. -* Operational assistance: Inspect a cluster's health or list topics during incidents. -* Onboarding and demos: Let team members issue high-level requests without memorizing every CLI flag. - -== How it works - -. Authenticate to Redpanda Cloud and receive a token using the xref:reference:rpk/rpk-cloud/rpk-cloud-login.adoc[`rpk cloud login` command]. -. Configure your MCP client using the `rpk cloud mcp install` command. Your client then starts the server on-demand using `rpk cloud mcp stdio`, authenticating with the Redpanda Cloud token from `rpk cloud login`. -. Prompt your assistant to perform Redpanda operations. The MCP server executes them in your Redpanda Cloud account using your Redpanda Cloud token. - -=== Components - -The Redpanda Cloud Management MCP Server requires these components: - -* AI client (Claude, Claude Code, or any other MCP client) that connects to the MCP server. -* Redpanda CLI (`rpk`) for obtaining a token and starting the MCP server. -* Redpanda Cloud account that the MCP server can connect to and issue API requests. - -== Security considerations - -MCP servers authenticate to Redpanda Cloud using your personal or service account credentials. However, there is **no auditing or access control** that distinguishes between actions performed by MCP servers versus direct API calls: - -* All API actions appear in Redpanda Cloud's internal logs as coming from the authenticated user account, not the specific MCP server. -* You cannot audit which MCP server performed which operations, as Redpanda Cloud logs are not accessible to users. -* You cannot restrict specific MCP servers to only certain API endpoints or resources. - -== Next steps - -* xref:ai-agents:mcp/quickstart.adoc[] -* xref:ai-agents:mcp/configuration.adoc[] - -TIP: The Redpanda documentation site has a read-only MCP server that provides access to Redpanda docs and examples. This server has no access to your Redpanda Cloud account or clusters. See xref:home:ROOT:mcp-setup.adoc[]. diff --git a/modules/ai-agents/pages/mcp/quickstart.adoc b/modules/ai-agents/pages/mcp/quickstart.adoc deleted file mode 100644 index d806d0d36..000000000 --- a/modules/ai-agents/pages/mcp/quickstart.adoc +++ /dev/null @@ -1,75 +0,0 @@ -= Redpanda Cloud Management MCP Server Quickstart -:page-aliases: mcp/local/quickstart.adoc -:page-beta: true -:description: Connect your Claude AI agent to your Redpanda Cloud account and clusters using the Redpanda Cloud Management MCP Server. -:page-topic-type: tutorial -:personas: agent_developer, platform_admin -// Reader journey: "I'm new" - seeking first hands-on experience -// Learning objectives - what readers will achieve by completing this quickstart: -:learning-objective-1: Authenticate to Redpanda Cloud with rpk -:learning-objective-2: Install the MCP integration for Claude -:learning-objective-3: Issue natural language commands to manage clusters - -In this quickstart, you'll get your Claude AI agent talking to Redpanda Cloud using the xref:ai-agents:mcp/overview.adoc[Redpanda Cloud Management MCP Server]. - -After completing this quickstart, you will be able to: - -* [ ] {learning-objective-1} -* [ ] {learning-objective-2} -* [ ] {learning-objective-3} - -== Prerequisites - -* At least version 25.2.3 of xref:manage:rpk/rpk-install.adoc[`rpk` installed on your computer] -* Access to a Redpanda Cloud account -* link:https://support.anthropic.com/en/articles/10065433-installing-claude-desktop[Claude] or link:https://docs.anthropic.com/en/docs/claude-code/setup[Claude Code] installed -+ -TIP: For other clients, see xref:ai-agents:mcp/configuration.adoc[]. - -== Set up the MCP server - -. Verify your `rpk` version. -+ -```bash -rpk version -``` -+ -Ensure the version is at least 25.2.3. - -. Log in to Redpanda Cloud. -+ -```bash -rpk cloud login -``` -+ -A browser window opens. Sign in to grant access. After you sign in, `rpk` stores a token locally. This token is not shared with your AI agent. It is used by the MCP server to authenticate requests to your Redpanda Cloud account. - -. Install the MCP integration. Choose one client: -+ -```bash -# Claude desktop -rpk cloud mcp install --client claude - -# Claude Code (IDE) -rpk cloud mcp install --client claude-code -``` -+ -This command configures the MCP server for your client. If you need to update the integration, re-run the install command for your client. - -== Start prompting - -Launch Claude or Claude Code and try one of these prompts: - -* “Create a Redpanda Cloud cluster named `dev-mcp`.” -* “List topics in `dev-mcp`.” -* “Create a topic `orders-raw` with 6 partitions.” - -:note-caption: Delete operations are opt-in - -NOTE: The MCP server does *not* expose API endpoints that result in delete operations by default. Use `--allow-delete` only if you intentionally want to enable delete operations. See xref:ai-agents:mcp/configuration.adoc#enable_delete_operations[Enable delete operations]. - -:note-caption: Note - -== Next steps - -* xref:ai-agents:mcp/configuration.adoc[] diff --git a/modules/ai-agents/pages/observability/concepts.adoc b/modules/ai-agents/pages/observability/concepts.adoc deleted file mode 100644 index 0f76864b0..000000000 --- a/modules/ai-agents/pages/observability/concepts.adoc +++ /dev/null @@ -1,339 +0,0 @@ -= Transcripts and AI Observability -:description: Understand how Redpanda captures end-to-end execution transcripts on an immutable distributed log for agent governance and observability. -:page-topic-type: concepts -:personas: evaluator, agent_developer, platform_admin, data_engineer -:learning-objective-1: Explain how transcripts and spans capture execution flow -:learning-objective-2: Interpret transcript structure for debugging and monitoring -:learning-objective-3: Distinguish between transcripts and audit logs - -include::ai-agents:partial$adp-la.adoc[] - -Redpanda provides complete observability and governance for AI agents through automated glossterm:transcript[] capture. Every agent execution, from simple tool calls to complex multi-agent, multi-turn workflows, generates a permanent, write-once record stored on Redpanda's glossterm:log[distributed log]. This captures all agent reasoning, tool invocations, model interactions, and data flows with 100% sampling and no gaps. - -With transcripts, organizations gain the ability to debug agent behavior, identify performance bottlenecks, meet regulatory compliance requirements, and maintain accountability for AI-driven decisions. Transcripts use OpenTelemetry standards and glossterm:Raft[]-based consensus for correctness, establishing a trustworthy foundation for agent governance. - -After reading this page, you will be able to: - -* [ ] {learning-objective-1} -* [ ] {learning-objective-2} -* [ ] {learning-objective-3} - -== What are transcripts - -A transcript records the complete execution of an agentic behavior from start to finish. It captures every step — across multiple agents, tools, models, and services — in a single, traceable record. The AI Gateway and every glossterm:AI agent[,agent] and glossterm:MCP server[] in your Agentic Data Plane (ADP) automatically emit OpenTelemetry traces to a glossterm:topic[] called `redpanda.otel_traces`. Redpanda's immutable distributed log stores these traces. - -Transcripts capture: - -* Tool invocations and results -* Agent reasoning steps -* Data processing operations -* External API calls -* Error conditions -* Performance metrics - -With 100% sampling, every operation is captured with no gaps. The underlying storage uses a distributed log built on Raft consensus (with TLA+ proven correctness), giving transcripts a trustworthy, immutable record for governance, debugging, and performance analysis. - -== Traces and spans - -glossterm:OpenTelemetry[] traces provide a complete picture of how a request flows through your system: - -* A _trace_ represents the entire lifecycle of a request (for example, a tool invocation from start to finish). -* A _span_ represents a single unit of work within that trace (such as a data processing operation or an external API call). -* A trace contains one or more spans organized hierarchically, showing how operations relate to each other. - -== Agent transcript hierarchy - -Agent executions create a hierarchy of spans that reflect how agents process requests. Understanding this hierarchy helps you interpret agent behavior and identify where issues occur. - -=== Agent span types - -Agent transcripts contain these span types: - -[cols="2,3,3", options="header"] -|=== -| Span Type | Description | Use To - -| `ai-agent` -| Top-level span representing the entire agent invocation from start to finish. Includes all processing time, from receiving the request through executing the reasoning loop, calling tools, and returning the final response. -| Measure total request duration and identify slow agent invocations. - -| `agent` -| Internal agent processing that represents reasoning and decision-making. Shows time spent in the glossterm:large language model (LLM)[,LLM] reasoning loop, including context processing, tool selection, and response generation. Multiple `agent` spans may appear when the agent iterates through its reasoning loop. -| Track reasoning time and identify iteration patterns. - -| `invoke_agent` -| Agent and sub-agent invocation in multi-agent architectures, following the https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/[OpenTelemetry agent invocation semantic conventions^]. Represents one agent calling another via the glossterm:Agent2Agent (A2A) protocol[,A2A protocol]. -| Trace calls between root agents and sub-agents, measure cross-agent latency, and identify which sub-agent was invoked. - -| `openai`, `anthropic`, or other LLM providers -| LLM provider API call showing calls to the language model. The span name matches the provider, and attributes typically include the model name (like `gpt-5.2` or `claude-sonnet-4-5`). -| Identify which model was called, measure LLM response time, and debug LLM API errors. - -| `rpcn-mcp` -| MCP tool invocation representing calls to MCP servers. Shows tool execution time, including network latency and tool processing. -| Measure tool execution time and identify slow MCP tool calls. -|=== - -=== Typical agent execution flow - -A simple agent request creates this hierarchy: - ----- -ai-agent (6.65 seconds) -├── agent (6.41 seconds) -│ ├── invoke_agent: customer-support-agent (6.39 seconds) -│ │ └── openai: chat gpt-5.2 (6.2 seconds) ----- - -This hierarchy shows that the LLM API call (6.2 seconds) accounts for most of the total agent invocation time (6.65 seconds), revealing the bottleneck in this execution flow. - -== MCP server transcript hierarchy - -MCP server tool invocations produce a different span hierarchy focused on tool execution and internal processing. This structure reveals performance bottlenecks and helps debug tool-specific issues. - -=== MCP server span types - -MCP server transcripts contain these span types: - -[cols="2,3,3", options="header"] -|=== -| Span Type | Description | Use To - -| `mcp-\{server-id\}` -| Top-level span representing the entire MCP server invocation. The server ID uniquely identifies the MCP server instance. This span encompasses all tool execution from request receipt to response completion. -| Measure total MCP server response time and identify slow tool invocations. - -| `service` -| Internal service processing span that appears at multiple levels in the hierarchy. Represents Redpanda Connect service operations including routing, processing, and component execution. -| Track internal processing overhead and identify where time is spent in the service layer. - -| Tool name (for example, `get_order_status`, `get_customer_history`) -| The specific MCP tool being invoked. This span name matches the tool name defined in the MCP server configuration. -| Identify which tool was called and measure tool-specific execution time. - -| `processors` -| Processor pipeline execution span showing the collection of processors that process the tool's data. Appears as a child of the tool invocation span. -| Measure total processor pipeline execution time. - -| Processor name (for example, `mapping`, `http`, `branch`) -| Individual processor execution span representing a single Redpanda Connect processor. The span name matches the processor type. -| Identify slow processors and debug processing logic. -|=== - -=== Typical MCP server execution flow - -An MCP tool invocation creates this hierarchy: - ----- -mcp-d5mnvn251oos73 (4.00 seconds) -├── service > get_order_status (4.07 seconds) -│ └── service > processors (43 microseconds) -│ └── service > mapping (18 microseconds) ----- - -This shows: - -1. Total MCP server invocation: 4.00 seconds -2. Tool execution (get_order_status): 4.07 seconds -3. Processor pipeline: 43 microseconds -4. Mapping processor: 18 microseconds (data transformation) - -The majority of time (4+ seconds) is spent in tool execution, while internal processing (mapping) takes only microseconds. This indicates the tool itself (likely making external API calls or database queries) is the bottleneck, not Redpanda Connect's internal processing. - -== Transcript layers and scope - -Transcripts contain multiple layers of instrumentation, from HTTP transport through application logic to external service calls. The `scope.name` field in each span identifies which instrumentation layer created that span. - -=== Instrumentation layers - -A complete agent transcript includes these layers: - -[cols="2,2,4", options="header"] -|=== -| Layer | Scope Name | Purpose - -| HTTP Server -| `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` -| HTTP transport layer receiving requests. Shows request/response sizes, status codes, client addresses, and network details. - -| AI SDK (Agent) -| `github.com/redpanda-data/ai-sdk-go/plugins/otel` -| Agent application logic. Shows agent invocations, LLM calls, tool executions, conversation IDs, token usage, and model details. Includes `gen_ai.*` semantic convention attributes. - -| HTTP Client -| `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` -| Outbound HTTP calls from agent to MCP servers. Shows target URLs, request methods, and response codes. - -| MCP Server -| `rpcn-mcp` -| MCP server tool execution. Shows tool name, input parameters, result size, and execution time. Appears as a separate `service.name` in resource attributes. - -| Redpanda Connect -| `redpanda-connect` -| Internal Redpanda Connect component execution within MCP tools. Shows pipeline and individual component spans. -|=== - -=== How layers connect - -Layers connect through parent-child relationships in a single transcript: - ----- -ai-agent-http-server (HTTP Server layer) -└── invoke_agent customer-support-agent (AI SDK layer) - ├── chat gpt-5-nano (AI SDK layer, LLM call 1) - ├── execute_tool get_order_status (AI SDK layer) - │ └── HTTP POST (HTTP Client layer) - │ └── get_order_status (MCP Server layer, different service) - │ └── processors (Redpanda Connect layer) - └── chat gpt-5-nano (AI SDK layer, LLM call 2) ----- - -The request flow demonstrates: - -1. HTTP request arrives at agent -2. Agent invokes sub-agent -3. Agent makes first LLM call to decide what to do -4. Agent executes tool, making HTTP call to MCP server -5. MCP server processes tool through its pipeline -6. Agent makes second LLM call with tool results -7. Response returns through HTTP layer - -=== Cross-service transcripts - -When agents call MCP tools, the transcript spans multiple services. Each service has a different `service.name` in the resource attributes: - -* Agent spans: `"service.name": "ai-agent"` -* MCP server spans: `"service.name": "mcp-\{server-id\}"` - -Both use the same `traceId`, allowing you to follow a request across service boundaries. - -=== Key attributes by layer - -Different layers expose different attributes: - -HTTP Server/Client layer (following https://opentelemetry.io/docs/specs/semconv/http/http-spans/[OpenTelemetry semantic conventions for HTTP^]): - -- `http.request.method`, `http.response.status_code` -- `server.address`, `url.path`, `url.full` -- `network.peer.address`, `network.peer.port` -- `http.request.body.size`, `http.response.body.size` - -AI SDK layer (following https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/[OpenTelemetry semantic conventions for generative AI^]): - -- `gen_ai.operation.name`: Operation type (`invoke_agent`, `chat`, `execute_tool`) -- `gen_ai.conversation.id`: Links spans to the same conversation session. A conversation may include multiple agent invocations (one per user request). Each invocation creates a separate trace that shares the same conversation ID. -- `gen_ai.agent.name`: Sub-agent name for multi-agent systems -- `gen_ai.provider.name`, `gen_ai.request.model`: LLM provider and model -- `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`: Token consumption -- `gen_ai.tool.name`, `gen_ai.tool.call.arguments`: Tool execution details -- `gen_ai.input.messages`, `gen_ai.output.messages`: Full LLM conversation context - -MCP Server layer: - -- Tool-specific attributes like `order_id`, `customer_id` -- `result_prefix`, `result_length`: Tool result metadata - -Redpanda Connect layer: - -- Component-specific attributes from your tool configuration - -The `scope.name` field identifies which instrumentation layer created each span. - -== Understand the transcript structure - -Each span captures a unit of work. Here's what a typical MCP tool invocation looks like: - -[,json] ----- -{ - "traceId": "71cad555b35602fbb35f035d6114db54", - "spanId": "43ad6bc31a826afd", - "name": "http_processor", - "attributes": [ - {"key": "city_name", "value": {"stringValue": "london"}}, - {"key": "result_length", "value": {"intValue": "198"}} - ], - "startTimeUnixNano": "1765198415253280028", - "endTimeUnixNano": "1765198424660663434", - "instrumentationScope": {"name": "rpcn-mcp"}, - "status": {"code": 0, "message": ""} -} ----- - -* `traceId` links all spans in the same request across services -* `spanId` uniquely identifies this span -* `name` identifies the operation or tool -* `instrumentationScope.name` identifies which layer created the span (`rpcn-mcp` for MCP tools, `redpanda-connect` for internal processing) -* `attributes` contain operation-specific metadata -* `status.code` indicates success (0) or error (2) - -=== Parent-child relationships - -Transcripts show how operations relate. A tool invocation (parent) may trigger internal operations (children): - -[,json] ----- -{ - "traceId": "71cad555b35602fbb35f035d6114db54", - "spanId": "ed45544a7d7b08d4", - "parentSpanId": "43ad6bc31a826afd", - "name": "http", - "instrumentationScope": {"name": "redpanda-connect"}, - "status": {"code": 0, "message": ""} -} ----- - -The `parentSpanId` links this child span to the parent tool invocation. Both share the same `traceId` so you can reconstruct the complete operation. - -== Error events in transcripts - -When something goes wrong, transcripts capture error details: - -[,json] ----- -{ - "traceId": "71cad555b35602fbb35f035d6114db54", - "spanId": "ba332199f3af6d7f", - "parentSpanId": "43ad6bc31a826afd", - "name": "http_request", - "events": [ - { - "name": "event", - "timeUnixNano": "1765198420254169629", - "attributes": [{"key": "error", "value": {"stringValue": "type"}}] - } - ], - "status": {"code": 0, "message": ""} -} ----- - -The `events` array captures what happened and when. Use `timeUnixNano` to see exactly when the error occurred within the operation. - -[[opentelemetry-traces-topic]] -== How Redpanda stores trace data - -The `redpanda.otel_traces` topic stores OpenTelemetry spans using Redpanda's glossterm:Schema Registry[] wire format, with a custom Protobuf schema named `redpanda.otel_traces-value` that follows the https://opentelemetry.io/docs/specs/otel/protocol/[OpenTelemetry Protocol (OTLP)^] specification. Spans include attributes following OpenTelemetry https://opentelemetry.io/docs/specs/semconv/gen-ai/[semantic conventions for generative AI^], such as `gen_ai.operation.name` and `gen_ai.conversation.id`. The schema is automatically registered in the Schema Registry with the topic, so Kafka clients can consume and deserialize trace data correctly. - -Redpanda manages both the `redpanda.otel_traces` topic and its schema automatically. If you delete either the topic or the schema, they are recreated automatically. However, deleting the topic permanently deletes all trace data, and the topic comes back empty. Do not produce your own data to this topic. It is reserved for OpenTelemetry traces. - -=== Topic configuration and lifecycle - -The `redpanda.otel_traces` topic has a predefined retention policy. Configuration changes to this topic are not supported. If you modify settings, Redpanda reverts them to the default values. - -The topic persists in your cluster even after all agents and MCP servers are deleted, allowing you to retain historical trace data for analysis. - -Transcripts may contain sensitive information from your tool inputs and outputs. Consider implementing appropriate glossterm:ACL[access control lists (ACLs)] for the `redpanda.otel_traces` topic, and review the data in transcripts before sharing or exporting to external systems. - -== Transcripts compared to audit logs - -Transcripts and audit logs serve different but complementary purposes. - -Transcripts provide: - -* A complete, immutable record of every execution step, stored on Redpanda's distributed log with no gaps -* Hierarchical view of request flow through your system (parent-child span relationships) -* Detailed timing information for performance analysis -* Ability to reconstruct execution paths and identify bottlenecks - -Transcripts are optimized for execution-level observability and governance. For user-level accountability tracking ("who initiated what"), use the session and task topics for agents, which provide records of agent conversations and task execution. - diff --git a/modules/ai-agents/pages/observability/index.adoc b/modules/ai-agents/pages/observability/index.adoc deleted file mode 100644 index 2b10f3b63..000000000 --- a/modules/ai-agents/pages/observability/index.adoc +++ /dev/null @@ -1,7 +0,0 @@ -= Transcripts -:page-layout: index -:description: Govern agentic AI with complete execution transcripts built on Redpanda's immutable distributed log. - -include::ai-agents:partial$adp-la.adoc[] - -{description} diff --git a/modules/ai-agents/partials/adp-la.adoc b/modules/ai-agents/partials/adp-la.adoc deleted file mode 100644 index c2c6c0c2d..000000000 --- a/modules/ai-agents/partials/adp-la.adoc +++ /dev/null @@ -1 +0,0 @@ -IMPORTANT: Redpanda Agentic Data Plane is supported only on BYOC clusters running with AWS and Redpanda version 25.3+. It is currently in glossterm:LA[, limited availability]. diff --git a/modules/ai-agents/partials/byoc-aws-requirement.adoc b/modules/ai-agents/partials/byoc-aws-requirement.adoc deleted file mode 100644 index 86fdf86a6..000000000 --- a/modules/ai-agents/partials/byoc-aws-requirement.adoc +++ /dev/null @@ -1 +0,0 @@ -NOTE: The Agentic Data Plane is supported on BYOC clusters running with AWS and Redpanda version 25.3 and later. diff --git a/modules/get-started/pages/cloud-overview.adoc b/modules/get-started/pages/cloud-overview.adoc index 0bb4a6626..bf83f196f 100644 --- a/modules/get-started/pages/cloud-overview.adoc +++ b/modules/get-started/pages/cloud-overview.adoc @@ -15,10 +15,6 @@ Redpanda ADP includes the following key components: * *Transcripts*: End-to-end execution records built on an immutable log with formal correctness guarantees. Transcripts are the keystone of agent governance. * *AI Gateway*: High-availability model routing with fiscal controls and per-tenant cost attribution across LLM providers. -For more information, see xref:ai-agents:adp-overview.adoc[Redpanda Agentic Data Plane Overview]. - -include::ai-agents:partial$adp-la.adoc[] - == Redpanda Cloud deployment options Redpanda Cloud applications are supported by three fully-managed deployment options: @@ -43,7 +39,7 @@ Redpanda Cloud applications are supported by three fully-managed deployment opti | *Deployment* | Redpanda's cloud (AWS/GCP) -| Redpanda's cloud (AWS/Azure/GCP)footnote:dedicated-azure-la[Dedicated on Azure is in glossterm:LA[, limited availability].] +| Redpanda's cloud (AWS/Azure/GCP) | Your cloud account (AWS/Azure/GCP) | *Redpanda ADP* @@ -141,7 +137,7 @@ Serverless is the fastest and easiest way to start data streaming. With Serverle [NOTE] ==== -* Serverless on GCP is in a glossterm:beta[] release. +* Serverless on GCP is currently in a glossterm:beta[] release. ==== ==== Sign up for Serverless @@ -166,16 +162,9 @@ Redpanda creates a cloud organization for you and sends you a welcome email. You AWS Marketplace:: + -- -New subscriptions to Redpanda Cloud through xref:billing:aws-pay-as-you-go.adoc[AWS Marketplace] receive $300 (USD) in free credits to spend in the first 30 days. AWS Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. - -Redpanda creates a cloud organization for you and sends you a welcome email. --- -Google Cloud Marketplace:: -+ --- -New subscriptions to Redpanda Cloud through xref:billing:gcp-pay-as-you-go.adoc[Google Cloud Marketplace] receive $300 (USD) in free credits to spend in the first 30 days. Google Cloud Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. +New subscriptions to Redpanda Cloud through xref:billing:aws-pay-as-you-go.adoc[AWS Marketplace] receive $300 (USD) in free credits to spend in the first 30 days. AWS Marketplace charges for anything beyond $300, unless you cancel the subscription. After your free credits have been used, you can continue using your cluster without any commitment, only paying for what you consume and canceling anytime. -Redpanda creates a cloud organization for you and sends you a welcome email. +Redpanda creates a cloud organization for you and sends you a welcome email. -- ===== @@ -384,7 +373,7 @@ Because Redpanda Cloud is a fully-managed service that provides maintenance, dat New clusters in Redpanda Cloud generally include functionality added in Self-Managed versions immediately. Existing clusters include new functionality when they get upgraded to the latest version. -Redpanda Cloud deployments do not support the following functionality available in Redpanda Streaming deployments: +Redpanda Cloud deployments do not support the following functionality available in Redpanda Self-Managed deployments: - Kafka API OIDC authentication. However, Redpanda Cloud does support xref:security:cloud-authentication.adoc#single-sign-on[SSO to the Redpanda Cloud UI]. - Admin API. @@ -420,8 +409,8 @@ Features in limited availability are production-ready and are covered by Redpand The following features are currently in limited availability in Redpanda Cloud: -* xref:ai-agents:adp-overview.adoc[Redpanda Agentic Data Plane (ADP)] -* xref:get-started:cluster-types/create-dedicated-cloud-cluster.adoc[Dedicated for Azure] +* Redpanda ADP including AI agents, AI Gateway, and transcripts +* Dedicated for Azure == Features in beta @@ -429,19 +418,15 @@ Features in beta are available for testing and feedback. They are not covered by The following features are currently in beta in Redpanda Cloud: -* xref:get-started:cluster-types/serverless.adoc#get-started-with-serverless[Serverless on GCP] -* xref:get-started:cluster-types/byoc/azure/vnet-azure.adoc[BYOVNet on Azure] -* xref:get-started:cluster-types/byoc/gcp/enable-secrets-byovpc-gcp.adoc[Secrets management for BYOVPC on GCP] -* xref:ai-agents:mcp/overview.adoc[Redpanda Cloud Management MCP Server] -* xref:manage:iceberg/iceberg-topics-aws-glue.adoc[Querying Iceberg topics with AWS Glue] -* xref:develop:managed-connectors/converters-and-serialization.adoc#protobuf-converter[Protobuf converter for managed connectors] +* BYOVNet for Azure +* Secrets management for BYOVPC on GCP +* Several Redpanda Connect components include::shared:partial$suggested-video.adoc[] * https://www.youtube.com/watch?v=gVlzsJAYT64&ab_channel=RedpandaData[YouTube - What is Redpanda BYOC? (3 mins)^] == Next steps -* xref:ai-agents:index.adoc[Build AI agents with Redpanda ADP] * xref:manage:maintenance.adoc[Learn about upgrades and maintenance] * xref:get-started:cluster-types/serverless.adoc[Create a Serverless cluster] * xref:get-started:cluster-types/byoc/index.adoc[Create a BYOC cluster] diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc index 973af7feb..0b00e205c 100644 --- a/modules/get-started/pages/whats-new-cloud.adoc +++ b/modules/get-started/pages/whats-new-cloud.adoc @@ -199,9 +199,6 @@ Redpanda Cloud now sends email notifications to organization admins when credit === Agentic Data Plane (ADP): LA Redpanda Agentic Data Plane (ADP) is now available in glossterm:LA[, limited availability] (LA). Redpanda ADP provides enterprise-grade infrastructure for building, deploying, and governing AI agents at scale. Key capabilities include declarative agents, MCP servers backed by 300+ connectors, an AI Gateway with model failover and fiscal controls, and compliance-grade transcripts built on Redpanda's immutable log. - -See xref:ai-agents:adp-overview.adoc[Agentic Data Plane Overview]. - === Serverless on AWS: GA xref:get-started:cluster-types/serverless.adoc[Serverless] on AWS is now generally available (GA). This release includes private networking with AWS PrivateLink. You can use the Cloud Console, the Cloud API, or the Redpanda Terraform provider to create and manage Serverless private links. Serverless is the easiest and fastest way to begin streaming data with Redpanda. @@ -333,12 +330,14 @@ Enable multi-factor authentication (MFA) to add an extra layer of security to yo === Redpanda Cloud Management MCP Server: beta -Connect AI assistants like Claude directly to your Redpanda Cloud account with the new xref:ai-agents:mcp/overview.adoc[Redpanda Cloud Management MCP Server]. This server runs on your computer and provides AI tools for managing clusters, topics, and other cloud resources through natural language commands. +Connect AI assistants like Claude directly to your Redpanda Cloud account with the new xref:agentic-data-plane:mcp:overview.adoc[Redpanda Cloud Management MCP Server]. This server runs on your computer and provides AI tools for managing clusters, topics, and other cloud resources through natural language commands. -Ask your AI assistant to "Create a new topic called user-events" or "List all clusters in my account" and it will handle the technical details automatically. Get started with the xref:ai-agents:mcp/quickstart.adoc[quickstart guide]. +Ask your AI assistant to "Create a new topic called user-events" or "List all clusters in my account" and it will handle the technical details automatically. Get started with the xref:agentic-data-plane:mcp:create-server.adoc[Create a Server guide]. The Redpanda Cloud Management MCP Server uses the Model Context Protocol (MCP) to extend AI assistants with Redpanda-specific capabilities, making cloud operations more accessible through conversational interfaces. + + === Automatic topic creation and topic limit For BYOC and Dedicated clusters, you can now configure the `auto_create_topics_enabled` cluster property to automatically create a topic if a client produces to a non-existent topic. diff --git a/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp-install.adoc b/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp-install.adoc deleted file mode 100644 index bd49810e1..000000000 --- a/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp-install.adoc +++ /dev/null @@ -1,3 +0,0 @@ -= rpk cloud mcp install - -include::streaming:reference:partial$rpk-cloud/rpk-cloud-mcp-install.adoc[tag=single-source] diff --git a/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp-stdio.adoc b/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp-stdio.adoc deleted file mode 100644 index 817724f86..000000000 --- a/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp-stdio.adoc +++ /dev/null @@ -1,3 +0,0 @@ -= rpk cloud mcp stdio - -include::streaming:reference:partial$rpk-cloud/rpk-cloud-mcp-stdio.adoc[tag=single-source] diff --git a/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp.adoc b/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp.adoc deleted file mode 100644 index 6ba0878a1..000000000 --- a/modules/reference/pages/rpk/rpk-cloud/rpk-cloud-mcp.adoc +++ /dev/null @@ -1,3 +0,0 @@ -= rpk cloud mcp - -include::streaming:reference:partial$rpk-cloud/rpk-cloud-mcp.adoc[tag=single-source]