Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased

### Dashboard — self-hosted (enterprise) mount readiness

The engine can now be mounted in any Rails app as the self-hosted
observability surface (see `docs/framework/self-hosted-observability.md`):

- **One install generator**: the duplicate `active_agent:dashboard:install`
variant that copied eight migrations (agents, sandboxes, recordings —
tables for models with no shipped controllers or routes) is removed.
The surviving generator installs the telemetry traces table only and
gains `--skip_migrations` / `--skip_routes`; its initializer template now
covers authentication, `ingest_api_key`, and multi-tenant options.
- **Canonical mount path is `/activeagents`** (generator, dummy app and
docs updated), and the telemetry client now derives its local ingest
endpoint from wherever the engine is actually mounted — any mount path,
including `/` on a dedicated subdomain, works.
`Telemetry::Configuration::LOCAL_ENDPOINT_PATH` remains as the fallback
when the engine isn't mounted.
- **`TracesController` honors configuration**: index/metrics/time-series
queries now go through `ActiveAgent::Dashboard.trace_model` (previously
only `show` did) and are scoped with `for_account(current_owner)`, so a
`trace_model_class` override and multi-tenant scoping apply everywhere.
- **Single-tenant ingest auth**: new `config.ingest_api_key` requires a
matching Bearer token on `POST <mount>/api/traces` when set. The
telemetry reporter and ruby_llm_telemetry already send their `api_key`
as a Bearer header, so remote apps need no changes.
- **Metrics page no longer 500s with data**: the per-agent stats table
read a grouped SQL alias through a model method that expected per-trace
token columns.
- Removed the never-consumed `base_controller_class` config attribute.

### Dashboard & Telemetry — dev console readiness

The dashboard engine — Active Agent's local dev console — now works out of
Expand Down
4 changes: 0 additions & 4 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,4 @@ source "https://rubygems.org"
gem "debug" unless ENV["CI"] == "true"
gem "rubocop-rails-omakase"

# Until activeagents-telemetry is published to RubyGems, resolve the gemspec
# dependency from GitHub. Remove this line after the first gem push.
gem "activeagents-telemetry", github: "activeagents/activeagents-telemetry"

gemspec
2 changes: 2 additions & 0 deletions docs/.vitepress/config.mts
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,8 @@ export default defineConfig({
{ text: 'Configuration', link: '/framework/configuration' },
{ text: 'Instrumentation', link: '/framework/instrumentation' },
{ text: 'Telemetry', link: '/framework/telemetry' },
{ text: 'Dashboard (Dev Console)', link: '/framework/dashboard' },
{ text: 'Self-Hosted Observability', link: '/framework/self-hosted-observability' },
{ text: 'Retries', link: '/framework/retries' },
{ text: 'Rails Integration', link: '/framework/rails' },
{ text: 'Testing', link: '/framework/testing' },
Expand Down
32 changes: 20 additions & 12 deletions docs/framework/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,15 @@ rails db:migrate

The generator:

- copies the `active_agent_telemetry_traces` migration (plus agent,
run, template, sandbox and recording tables for the full install),
- mounts the engine at `/active_agent`,
- copies the `active_agent_telemetry_traces` migration (the one table the
dashboard reads),
- mounts the engine at `/activeagents`,
- writes `config/initializers/active_agent_dashboard.rb`.

Deploying this beyond your laptop — for a team, or as the trace sink for
a fleet of apps? See
[Self-Hosted Observability](/framework/self-hosted-observability).

Then enable telemetry with local storage in `config/active_agent.yml`:

```yaml
Expand All @@ -34,18 +38,18 @@ telemetry:
local_storage: true
```

That's it. Run any agent and open `/active_agent` — each generation
That's it. Run any agent and open `/activeagents` — each generation
appears as a trace with prompt/LLM/tool spans, timing, token usage
(input / output / thinking), provider and model.

## What you get

| Page | Path | Contents |
|------|------|----------|
| Traces | `/active_agent/traces` | Every generation: agent + action, status, duration, tokens; expandable span timeline; All/Errors filter; 30s auto-refresh |
| Trace detail | `/active_agent/traces/:id` | Span waterfall with relative offsets, token breakdown, error details, raw payload |
| Metrics | `/active_agent/traces/metrics` | Last-24h totals: traces, tokens, avg duration, error rate, active agents; per-agent statistics |
| Ingest API | `POST /active_agent/api/traces` | JSON trace ingestion (used by `local_storage` mode and remote SDKs) |
| Traces | `/activeagents/traces` | Every generation: agent + action, status, duration, tokens; expandable span timeline; All/Errors filter; 30s auto-refresh |
| Trace detail | `/activeagents/traces/:id` | Span waterfall with relative offsets, token breakdown, error details, raw payload |
| Metrics | `/activeagents/traces/metrics` | Last-24h totals: traces, tokens, avg duration, error rate, active agents; per-agent statistics |
| Ingest API | `POST /activeagents/api/traces` | JSON trace ingestion (used by `local_storage` mode and remote SDKs) |

Time-series charts on the metrics page use the optional
[groupdate](https://github.com/ankane/groupdate) gem when present and
Expand All @@ -70,13 +74,17 @@ Or constrain the mount in `config/routes.rb`:

```ruby
authenticate :user, ->(u) { u.admin? } do
mount ActiveAgent::Dashboard::Engine => "/active_agent"
mount ActiveAgent::Dashboard::Engine => "/activeagents"
end
```

The local ingest endpoint is unauthenticated in local mode by design (it
receives traces from your own app process). In multi-tenant mode it
requires a Bearer token (see below).
The local ingest endpoint accepts unauthenticated posts by default (it
receives traces from your own app process on your own machine). If the
mount is reachable from other machines, set `config.ingest_api_key` to
require a Bearer token — see
[Self-Hosted Observability](/framework/self-hosted-observability). In
multi-tenant mode ingest always authenticates per-account keys (see
below).

## Sending traces to a remote endpoint instead

Expand Down
225 changes: 225 additions & 0 deletions docs/framework/self-hosted-observability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
# Self-Hosted Observability (Enterprise)

Run the Active Agent dashboard inside your own Rails app: traces, span
waterfalls and metrics served from your database, on your domain. Data
never leaves your infrastructure.

This is the same engine that powers the [dev console](/framework/dashboard)
— this guide covers deploying it as a shared, production observability
surface for a team or a fleet of apps. If you just want traces while you
build on your laptop, the dev console quick start is all you need.

## Two ways to run observability

| | Self-hosted engine | Hosted platform (activeagents.ai) |
|---|---|---|
| Where data lives | Your database | Your workspace on the platform |
| Setup | Mount the engine in a Rails app | Point telemetry at an API key |
| Traces + span waterfall | ✓ | ✓ |
| Metrics (24h aggregates, per-agent stats) | ✓ | ✓ |
| Ingest API for remote apps | ✓ (`<mount>/api/traces`) | ✓ (`https://api.activeagents.ai/v1/traces`) |
| Interactions (tool-call conversations), evaluations, scorecards, cost estimates | — | ✓ |
| Retention policies, team workspaces, plans | Yours to operate | ✓ |

The wire format is identical, so the choice is per-environment, not
per-app: the same `config/active_agent.yml` switches between them (see
[Getting traces in](#getting-traces-in)).

## Install

```ruby
# Gemfile
gem "activeagent"
```

```bash
bundle install
rails generate active_agent:dashboard:install
rails db:migrate
```

The generator creates exactly three things:

- `db/migrate/*_create_active_agent_telemetry_traces.rb` — the one table
the dashboard reads,
- `mount ActiveAgent::Dashboard::Engine => "/activeagents"` in
`config/routes.rb`,
- `config/initializers/active_agent_dashboard.rb` — authentication,
ingest key and multi-tenant options, commented.

Open `http://localhost:3000/activeagents` and you have the dashboard.
(`--skip_migrations` / `--skip_routes` are available if you manage either
yourself.)

## Routing: a path or a subdomain

The mount path is yours to choose — the telemetry client derives its
local ingest endpoint from wherever the engine is actually mounted, so
nothing else needs configuring:

```ruby
# A path on your main app:
mount ActiveAgent::Dashboard::Engine => "/activeagents"

# Or the root of a dedicated subdomain, e.g. activeagents.combinaut.com:
constraints subdomain: "activeagents" do
mount ActiveAgent::Dashboard::Engine => "/", as: :active_agent_subdomain
end
```

With the subdomain mount, the dashboard lives at
`https://activeagents.combinaut.com/` and remote apps post traces to
`https://activeagents.combinaut.com/api/traces`.

## Authentication (required in production)

Traces contain prompts, outputs and error messages. Without an
`authentication_method` the dashboard refuses to serve in production
(HTTP 403), so set one before deploying:

```ruby
# config/initializers/active_agent_dashboard.rb
ActiveAgent::Dashboard.configure do |config|
# Basic auth:
config.authentication_method = ->(controller) {
controller.authenticate_or_request_with_http_basic do |username, password|
username == "ops" && password == Rails.application.credentials.dashboard_password
end
}
# ...or Devise: ->(controller) { controller.authenticate_admin! }
end
```

The ingest API authenticates separately. In single-tenant mode it accepts
unauthenticated posts by default (fine for same-app `local_storage`, not
for a network-reachable mount) — set an ingest key whenever other
machines can reach it:

```ruby
config.ingest_api_key = Rails.application.credentials.dig(:active_agent, :ingest_api_key)
```

Requests without a matching `Authorization: Bearer <key>` header get a
401. The telemetry reporter and `ruby_llm_telemetry` already send their
configured `api_key` as a Bearer header, so remote apps need no changes.

## Getting traces in

**Same app** — the app that mounts the dashboard stores its own traces
directly, no HTTP involved:

```yaml
# config/active_agent.yml
production:
telemetry:
enabled: true
local_storage: true
```

**Other ActiveAgent apps in your fleet** — point their telemetry at your
mount:

```yaml
production:
telemetry:
enabled: true
endpoint: https://activeagents.combinaut.com/api/traces
api_key: <%= Rails.application.credentials.dig(:active_agent, :ingest_api_key) %>
```

**The hosted platform instead** — same file, different endpoint; this is
what "cloud mode" is:

```yaml
production:
telemetry:
enabled: true
endpoint: https://api.activeagents.ai/v1/traces
api_key: <%= ENV["ACTIVEAGENTS_API_KEY"] %>
```

**Multi-tenant mode** — if your self-hosted install itself serves multiple
accounts, enable `config.multi_tenant` with `account_class` /
`current_account_method`; ingest then authenticates per-account
`telemetry_api_key` Bearer tokens and processes asynchronously via
`ActiveAgent::ProcessTelemetryTracesJob` (requires an Active Job backend),
and every dashboard query scopes to the current account. Most self-hosted
installs should leave this off.

## RubyLLM applications

Apps that use [RubyLLM](https://rubyllm.com) directly (no
`ActiveAgent::Base`) can report chats and tool calls to the same endpoint
with the
[activeagents-telemetry-ruby_llm](https://rubygems.org/gems/activeagents-telemetry-ruby_llm)
adapter:

```ruby
# Gemfile
gem "activeagents-telemetry-ruby_llm"
```

```ruby
# config/initializers/ruby_llm_telemetry.rb
ActiveAgents::Telemetry::RubyLLM.subscribe!(
endpoint: "https://activeagents.combinaut.com/api/traces",
api_key: Rails.application.credentials.dig(:active_agent, :ingest_api_key),
service_name: "billing-app"
)
```

Attribute traffic to named agents with `with_agent("SupportAgent", action: "respond") { ... }`
or an `agent_resolver:` lambda — otherwise traffic reports as
`RubyLLM::Chat`. See the bridge's README for content capture
(off by default) and turn semantics.

## Optional: conversation persistence with solid_agent

Telemetry gives you traces; [solid_agent](https://github.com/activeagents/solid_agent)
additionally persists conversations (contexts, messages, generations —
including tool calls with arguments and results) in your database:

```bash
rails generate solid_agent:install
rails db:migrate
```

Generations record the same `trace_id` the telemetry pipeline uses
(thread it via `prompt_options[:trace_id]`), so conversation rows and
dashboard traces correlate.

## Operations

- **Retention is yours.** Nothing prunes automatically. A recurring job
as simple as
`ActiveAgent::TelemetryTrace.where("created_at < ?", 30.days.ago).delete_all`
is enough.
- **Database portability.** The engine's queries are plain ActiveRecord —
SQLite, MySQL and PostgreSQL all work. (The hosted platform's richer
views use PostgreSQL-specific SQL; the engine deliberately doesn't.)
- **CDN assets.** The default layout loads Tailwind, Turbo and Stimulus
from public CDNs. On CSP-strict or air-gapped networks the pages render
unstyled (fully functional, but plain) — set `config.layout` to a
layout of your own that bundles those assets locally.
- **Time-series charts** on the metrics page light up when the optional
[groupdate](https://github.com/ankane/groupdate) gem is installed.
- **Sensitive content.** Prompt/output capture obeys the telemetry
`redact_attributes` configuration — see [Telemetry](/framework/telemetry).

## Troubleshooting

- **No traces appear** — telemetry is opt-in per environment: check
`enabled: true` (and `local_storage: true` for same-app storage) under
the *current* environment key in `config/active_agent.yml`.
- **403 in production** — set `config.authentication_method` (see above).
- **401 from ingest** — the poster's `api_key` doesn't match
`config.ingest_api_key` (single-tenant) or an account
`telemetry_api_key` (multi-tenant).
- **Metrics page has no chart** — install `groupdate`.

## Future work

Rendering agent conversations persisted by RubyLLM's `acts_as` schema or
solid_agent tables directly in the engine (DB-level detection of agent
implementations) is on the roadmap; today those render on the hosted
platform via telemetry.
6 changes: 6 additions & 0 deletions docs/framework/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,12 @@ at_exit { ActiveAgent::Telemetry.shutdown }

## Self-Hosting

The dashboard engine already implements this endpoint spec: mount it and
its ingest API (`<mount>/api/traces`) receives traces from any app in
your fleet — see
[Self-Hosted Observability](/framework/self-hosted-observability). The
requirements below are for building your own receiver instead.

### Endpoint Requirements

Your telemetry endpoint must accept POST requests with:
Expand Down
3 changes: 0 additions & 3 deletions gemfiles/anthropic_1.12.gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,4 @@
gem "sqlite3", "~> 2.0"

gemspec path: ".."

Check failure on line 11 in gemfiles/anthropic_1.12.gemfile

View workflow job for this annotation

GitHub Actions / lint

Layout/TrailingEmptyLines: 1 trailing blank lines detected.
# Until activeagents-telemetry is published to RubyGems, resolve the gemspec
# dependency from GitHub. Remove after the first gem push.
gem "activeagents-telemetry", github: "activeagents/activeagents-telemetry"
3 changes: 0 additions & 3 deletions gemfiles/anthropic_1.14.gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,4 @@
gem "sqlite3", "~> 2.0"

gemspec path: ".."

Check failure on line 11 in gemfiles/anthropic_1.14.gemfile

View workflow job for this annotation

GitHub Actions / lint

Layout/TrailingEmptyLines: 1 trailing blank lines detected.
# Until activeagents-telemetry is published to RubyGems, resolve the gemspec
# dependency from GitHub. Remove after the first gem push.
gem "activeagents-telemetry", github: "activeagents/activeagents-telemetry"
3 changes: 0 additions & 3 deletions gemfiles/anthropic_latest.gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,4 @@
gem "sqlite3", "~> 2.0"

gemspec path: ".."

Check failure on line 9 in gemfiles/anthropic_latest.gemfile

View workflow job for this annotation

GitHub Actions / lint

Layout/TrailingEmptyLines: 1 trailing blank lines detected.
# Until activeagents-telemetry is published to RubyGems, resolve the gemspec
# dependency from GitHub. Remove after the first gem push.
gem "activeagents-telemetry", github: "activeagents/activeagents-telemetry"
3 changes: 0 additions & 3 deletions gemfiles/openai_0.34.gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,4 @@
gem "sqlite3", "~> 2.0"

gemspec path: ".."

Check failure on line 11 in gemfiles/openai_0.34.gemfile

View workflow job for this annotation

GitHub Actions / lint

Layout/TrailingEmptyLines: 1 trailing blank lines detected.
# Until activeagents-telemetry is published to RubyGems, resolve the gemspec
# dependency from GitHub. Remove after the first gem push.
gem "activeagents-telemetry", github: "activeagents/activeagents-telemetry"
3 changes: 0 additions & 3 deletions gemfiles/openai_0.35.gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,4 @@
gem "sqlite3", "~> 2.0"

gemspec path: ".."

Check failure on line 11 in gemfiles/openai_0.35.gemfile

View workflow job for this annotation

GitHub Actions / lint

Layout/TrailingEmptyLines: 1 trailing blank lines detected.
# Until activeagents-telemetry is published to RubyGems, resolve the gemspec
# dependency from GitHub. Remove after the first gem push.
gem "activeagents-telemetry", github: "activeagents/activeagents-telemetry"
Loading
Loading