diff --git a/docs.json b/docs.json index b13b7f1ca..849783d52 100644 --- a/docs.json +++ b/docs.json @@ -438,6 +438,7 @@ "enterprise/index", "enterprise/enterprise-vs-oss", "enterprise/quick-start", + "enterprise/analytics", { "group": "K8s Install", "pages": [ diff --git a/enterprise/analytics.mdx b/enterprise/analytics.mdx new file mode 100644 index 000000000..8fd45f0cd --- /dev/null +++ b/enterprise/analytics.mdx @@ -0,0 +1,122 @@ +--- +title: Analytics +description: Get started with LLM observability and tracing in OpenHands Enterprise. +icon: rocket +--- + +This guide walks you through setting up and using Laminar for Analytics in OpenHands Enterprise. +You'll opt into Analytics and configure conversations to automatically send traces to Laminar. + +## Who This Is For + +This guide is for users who want to explore analytics on their OpenHands Enterprise conversations. + +### Why Laminar? + +[Laminar](https://laminar.sh/) is an open source observability platform for AI agents like OpenHands. + +Use Laminar to view your conversation traces including prompts, tool calls, and answers. A trace is a record of what your agent did. + +Laminar can help you see where the agent went wrong. From traces, you can create signals. A signal is a natural language instruction to extract structured data from traces. Use signals to analyze recurring behavior across traces. You can then create better situations for prompting and measure them in Laminar. You can also analyze and improve your skills. + +For example, you can view all conversation traces related to a specific skill. + +Laminar can help you answer the following questions: +- On a trace, did the agent do a good job using the skill? +- On another trace, did the agent do a bad job? + +For more information on evaluating skills, see [Evaluating Agent Skills](https://www.openhands.dev/blog/evaluating-agent-skills). + +### Prerequisites + +Before you begin, make sure you completed the [Quick Start guide](/enterprise/quick-start). + +## Enable Analytics + +You should see an **Analytics Configuration** section on the application configuration page. + +Check the **Enable Analytics** box to have the installer set up and configure Laminar for analytics. + +![Configure Analytics](./images/laminar-configure-analytics.png) + +## Deploy + +OpenHands will begin deploying. You can expect the deployment status to transition from +**Missing** to **Unavailable** to **Ready**. This typically takes 10-15 minutes. + +![Deployment in progress](./images/laminar-deploy-in-progress.png) + +Click **Details** next to the deployment status to monitor individual resources. Resources +shown in orange are still deploying -- wait until all resources are ready. + +![Deployment status details](./images/laminar-deployment-status-details.png) + +## Access Laminar UI + +Once the deployment status shows **Ready**, navigate to `https://analytics.app.`. + +Click the **Continue with Keycloak** button: + +![Laminar Keycloak Auth](./images/laminar-keycloak-auth.png) + +## Create a Laminar project + +![Laminar Create Project](./images/laminar-create-project.png) + +Once a project has been created, Laminar is ready to listen for traces. + +![Laminar Listen Traces](./images/laminar-listen-traces.png) + +## Create an ingest only API Key + +Important: Always use ingest API keys when deploying. + +Create a key with ther right permissions. Ingest only keys are recommended as they only have write access to write traces. They cannot be used to read data. + +![Configure Laminar Ingest Only Key](./images/laminar-ingest-only-key.png) + +## Set Laminar Project API Key to enable automatic conversation traces + +Set the ingest only key as the Laminar Project API Key in the Admin Console configuration: + +![Configure Laminar Project API Key](./images/laminar-configure-key.png) + +Click **Save config**. + +## Deploy Updated Configuration + +Deploy the config change after setting the Laminar Project API Key in the Admin Console. + +![Laminar Deploy Again](./images/laminar-deploy-again.png) + +Wait for the deployment to complete. + +## Start a conversation + +Navigate to the OpenHands UI at `https://app.`. Start a new conversation and try a prompt. + +![Start a Conversation](./images/laminar-openhands-conversation.png) + +Your conversations will now automatically send a trace to Laminar. + +![Laminar Trace](./images/laminar-trace.png) + + + +## Next Steps + + + + Get the most out of your AI coding agents with effective prompting techniques. + + + Reach out to the OpenHands team for deployment assistance or questions. + + + Explore the full OpenHands documentation for usage guides and features. + + diff --git a/enterprise/images/laminar-configure-analytics.png b/enterprise/images/laminar-configure-analytics.png new file mode 100644 index 000000000..bae25f5c9 Binary files /dev/null and b/enterprise/images/laminar-configure-analytics.png differ diff --git a/enterprise/images/laminar-configure-key.png b/enterprise/images/laminar-configure-key.png new file mode 100644 index 000000000..6f880399b Binary files /dev/null and b/enterprise/images/laminar-configure-key.png differ diff --git a/enterprise/images/laminar-create-project.png b/enterprise/images/laminar-create-project.png new file mode 100644 index 000000000..128b51111 Binary files /dev/null and b/enterprise/images/laminar-create-project.png differ diff --git a/enterprise/images/laminar-deploy-again.png b/enterprise/images/laminar-deploy-again.png new file mode 100644 index 000000000..700b723a7 Binary files /dev/null and b/enterprise/images/laminar-deploy-again.png differ diff --git a/enterprise/images/laminar-deploy-in-progress.png b/enterprise/images/laminar-deploy-in-progress.png new file mode 100644 index 000000000..d99eb859c Binary files /dev/null and b/enterprise/images/laminar-deploy-in-progress.png differ diff --git a/enterprise/images/laminar-deployment-status-details.png b/enterprise/images/laminar-deployment-status-details.png new file mode 100644 index 000000000..91a3fd488 Binary files /dev/null and b/enterprise/images/laminar-deployment-status-details.png differ diff --git a/enterprise/images/laminar-ingest-only-key.png b/enterprise/images/laminar-ingest-only-key.png new file mode 100644 index 000000000..c5528b943 Binary files /dev/null and b/enterprise/images/laminar-ingest-only-key.png differ diff --git a/enterprise/images/laminar-keycloak-auth.png b/enterprise/images/laminar-keycloak-auth.png new file mode 100644 index 000000000..706a5633f Binary files /dev/null and b/enterprise/images/laminar-keycloak-auth.png differ diff --git a/enterprise/images/laminar-listen-traces.png b/enterprise/images/laminar-listen-traces.png new file mode 100644 index 000000000..a4145fd5e Binary files /dev/null and b/enterprise/images/laminar-listen-traces.png differ diff --git a/enterprise/images/laminar-openhands-conversation.png b/enterprise/images/laminar-openhands-conversation.png new file mode 100644 index 000000000..d4501161b Binary files /dev/null and b/enterprise/images/laminar-openhands-conversation.png differ diff --git a/enterprise/images/laminar-trace.png b/enterprise/images/laminar-trace.png new file mode 100644 index 000000000..5fc8868e6 Binary files /dev/null and b/enterprise/images/laminar-trace.png differ