Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion build/agents/build-your-agent/evals.mdx
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
---
title: 'Evals'
sidebarTitle: 'Evals'

Check warning on line 3 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L3

Did you really mean 'Evals'?
description: 'Test and evaluate your AI Agents with scenario-based evaluations and reusable Checks'
---

<Info>
**Rollout Status**: Evals is rolling out progressively, starting with Enterprise customers. If you don't see this feature in your account yet, reach out to your account manager to discuss access.

Check warning on line 8 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L8

Did you really mean 'Evals'?
</Info>

The Evals section is your command center for testing and evaluating AI Agent performance. Located in the **Evaluate** tab (next to the Build and Use tabs) in the Agent builder, Evals lets you create test sets, define reusable Checks, run automated evaluations, and monitor live Agent quality — all without manual testing.
The Evals section is your command center for testing and evaluating AI performance. Located in the **Evaluate** tab in the Agent builder and on the Workforce canvas, Evals lets you create test sets, define reusable Checks, run automated evaluations, and monitor live Agent quality — all without manual testing.

Check warning on line 11 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L11

Did you really mean 'Evals'?

Check warning on line 11 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L11

Did you really mean 'Evals'?

![Evaluate tab showing the Evals sidebar (Test, Runs, Checks, Publish, Monitor) and a Monitor dashboard with overall score, total runs, and Checks breakdown](/images/agent/agent-evals.png)

Check warning on line 13 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L13

Did you really mean 'Evals'?

## What you can do with Evals

Check warning on line 15 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L15

Did you really mean 'Evals'?

<CardGroup cols={3}>
<Card title="Run tests" icon="flask-vial">
Expand All @@ -28,11 +28,11 @@

---

## Evals sections

Check warning on line 31 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L31

Did you really mean 'Evals'?

The Evals area has five sections, shown in the left sidebar of the Evaluate tab:

Check warning on line 33 in build/agents/build-your-agent/evals.mdx

View check run for this annotation

Mintlify / Mintlify Validation (relevanceai) - vale-spellcheck

build/agents/build-your-agent/evals.mdx#L33

Did you really mean 'Evals'?

- **Test** — Create and manage test sets. Each test set holds scenarios that simulate users; running a scenario produces a conversation with your Agent that gets scored by attached Checks.

Check warning on line 35 in build/agents/build-your-agent/evals.mdx

View workflow job for this annotation

GitHub Actions / Documentation Lint Checks

5 settings listed as bullet points — consider using a table instead so they're easier to scan. [technical: 5 consecutive bullet items matching **Key**: value or **Key** — value pattern]
- **Runs** — Past evaluation run results. Browse average scores, tasks evaluated, progress status, credit spend, and creation date for every run.
- **Checks** — The reusable set of evaluation criteria. Create a Check once, then attach it to scenarios, to Monitor dashboards, or to one-off evaluations of completed tasks.
- **Publish** — Choose which test sets must pass before your Agent can be published. Set a minimum pass rate and optionally block publishing on failure.
Expand Down Expand Up @@ -89,6 +89,7 @@
| **Tool** | Select the tool to check for |
| **Position** | Whether the tool was used anywhere, used first, or used last |
| **Comparison** | Check if the tool was used at least, exactly, or at most X times |
| **Node** | Optional. When evaluating a Workforce, use the node picker to scope this check to a specific graph node (Agent or Tool). If the same Agent appears more than once in the graph, each occurrence is listed separately with a connection label for disambiguation. Leave blank to check across the whole conversation. |
</Accordion>
</AccordionGroup>

Expand Down Expand Up @@ -148,6 +149,8 @@
- Provide a prompt describing what the Tool should return (a fake response is generated based on your prompt).
- In the **Advanced** dropdown, you can select a **Simulation model** to control which model generates the simulated response.

When running a **Workforce** scenario, simulations are configured per node instead of globally. Each Agent and Tool node in the Workforce graph appears as a separate entry, so you can assign different simulated responses to the same Tool depending on which node calls it. If the same Agent appears in multiple positions in the graph, each occurrence is listed separately with a connection label (for example, "Research Agent ← Coordinator") so you can identify which instance you're configuring and simulate each independently.

8. Click **Save test scenario** to save your configuration.

<Tip>
Expand Down Expand Up @@ -205,6 +208,20 @@

---

## Evaluating Workforces

The **Evaluate** tab is also available on the Workforce canvas. Workforce evals work the same way as individual Agent evals — you create test sets, define Checks, run scenarios, and view results — with two additions for multi-agent graphs.

### Tool Usage checks with node scoping

When creating a Tool Usage check in a Workforce eval, a **Node** picker appears alongside the standard check fields. Select a specific Agent or Tool node to scope the check to that node's tool calls only. If the same Agent appears more than once in the graph, each occurrence is listed separately with a connection label so you can target the right instance. Leave the node field blank to check for tool usage anywhere in the conversation.

### Per-node tool simulation

Tool simulations in Workforce scenarios target individual nodes rather than tools globally. Each Agent and Tool node in the Workforce graph is listed as a separate simulation entry, letting you assign different fake responses to the same Tool depending on which node calls it. Duplicate Agent nodes appear with connection labels — for example, "Research Agent ← Coordinator" — so you can configure each occurrence independently.

---

## Running evaluations

You can run an entire test set or an individual scenario from within a test set by clicking the **Run** button on either.
Expand Down Expand Up @@ -284,7 +301,7 @@

You also get:

- **Overall score timeseries** to spot regressions or improvements over time.

Check warning on line 304 in build/agents/build-your-agent/evals.mdx

View workflow job for this annotation

GitHub Actions / Documentation Lint Checks

4 features listed as bullet points — consider using cards instead so they stand out visually. [technical: 4 consecutive bullet items matching **Feature** pattern, use <CardGroup> with <Card> components]
- **Per-Check charts** so you can see which criteria are slipping.
- **Version markers** that line up score changes with Agent publishes.
- **A list of evaluation runs** with score, name, and a drill-in to the full conversation.
Expand Down
1 change: 1 addition & 0 deletions build/workforces/create-a-workforce.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,4 @@ Once you've created a Workforce, explore the rest of the build guides to refine
- [Add Tools](/build/workforces/build-an-ai-workforce/add-tools) — connect Tools directly within your workflow
- [Add conditions](/build/workforces/build-an-ai-workforce/add-conditions) — create routing rules for smarter workflows
- [Edge settings](/build/workforces/build-an-ai-workforce/edge-settings) — configure how connections between nodes behave
- [Evaluate your Workforce](/build/agents/build-your-agent/evals) — use Checks, test sets, and Tool simulations to test your multi-agent system
Loading