Skip to content

Commit a82941f

Browse files
hubert-marekhubert-marek
authored
Merge pull request #138 from agent-diff-bench/fixes-kdd
Example Notebooks
2 parents 76898ab + b3b490b commit a82941f

3 files changed

Lines changed: 707 additions & 3 deletions

File tree

‎README.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,16 @@
55
Run it locally (or deploy it). Agents call sandboxed replicas of APIs that behave like the real ones, and you get deterministic diffs of every state change — no external services, no side effects, no rate limits.
66

77
<p align="center">
8-
<a href="https://arxiv.org/abs/2602.11224">Paper (arXiv)</a> •
8+
<a href="https://arxiv.org/abs/2602.11224"><img src="https://img.shields.io/badge/arXiv-2602.11224-b31b1b.svg" alt="arXiv"></a>
9+
<a href="https://huggingface.co/datasets/hubertmarek/agent-diff-bench"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Dataset-yellow.svg" alt="HuggingFace"></a>
10+
<a href="https://app.primeintellect.ai/dashboard/environments/hubert-marek/agent-diff-bench"><img src="https://img.shields.io/badge/Prime%20Intellect-Run%20Evals-blue.svg" alt="Prime Intellect"></a>
11+
<a href="https://colab.research.google.com/github/agent-diff-bench/agent-diff/blob/main/examples/react_agent_benchmark.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"></a>
12+
</p>
13+
14+
<p align="center">
915
<a href="https://agentdiff.dev">Website</a> •
1016
<a href="https://agentdiff.mintlify.app/introduction">Docs</a> •
11-
<a href="https://huggingface.co/datasets/hubertmarek/agent-diff-bench">Dataset</a> •
12-
<a href="https://app.primeintellect.ai/dashboard/environments/hubert-marek/agent-diff-bench">Prime Intellect</a> •
17+
<a href="https://arxiv.org/abs/2602.11224">Paper</a> •
1318
<a href="mailto:hubert@uni.minerva.edu">Feedback</a>
1419
</p>
1520

@@ -168,6 +173,11 @@ The fastest way to run Agent-Diff evaluations is via **[Prime Intellect](https:/
168173

169174
Alternatively, run locally or self-hosted using the SDK (see [To run evaluations](#to-run-evaluations) below).
170175

176+
### Example Notebooks
177+
178+
- **[ReAct Agent (Paper)](examples/react_agent_benchmark.ipynb)** — Custom ReAct loop matching the paper methodology [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/agent-diff-bench/agent-diff/blob/main/examples/react_agent_benchmark.ipynb)
179+
- **[LangChain Agent](examples/langchain_agent_benchmark.ipynb)** — LangChain agent with tool calling [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/agent-diff-bench/agent-diff/blob/main/examples/langchain_agent_benchmark.ipynb)
180+
171181
**Resources:**
172182
- **Dataset**: [hubertmarek/agent-diff-bench](https://huggingface.co/datasets/hubertmarek/agent-diff-bench) — 224 tasks across all 4 services (80/20 train/test split)
173183
- **Prime Intellect**: [agent-diff-bench on Prime Lab](https://app.primeintellect.ai/dashboard/environments/hubert-marek/agent-diff-bench) — hosted evaluations & RL training

‎examples/langchain_agent_benchmark.ipynb‎

Lines changed: 298 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,298 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Agent-Diff Benchmark: LangChain Agent\n",
8+
"\n",
9+
"Run the [Agent-Diff benchmark](https://arxiv.org/abs/2602.11224) using LangChain's built-in agent with tool calling.\n",
10+
"\n",
11+
"Unlike the [ReAct notebook](react_agent_benchmark.ipynb) which uses a custom XML-tag loop, this notebook lets LangChain handle the agent loop via the model's native function-calling protocol. The `BashExecutorProxy` from the `agent-diff` SDK is wrapped as a LangChain tool.\n",
12+
"\n",
13+
"All 4 services (Box, Calendar, Linear, Slack) are evaluated across 224 tasks.\n",
14+
"\n",
15+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/agent-diff-bench/agent-diff/blob/main/examples/langchain_agent_benchmark.ipynb)\n",
16+
"\n",
17+
"**Links:** [Paper](https://arxiv.org/abs/2602.11224) | [Dataset](https://huggingface.co/datasets/hubertmarek/agent-diff-bench) | [GitHub](https://github.com/agent-diff-bench/agent-diff)"
18+
]
19+
},
20+
{
21+
"cell_type": "code",
22+
"execution_count": null,
23+
"metadata": {},
24+
"outputs": [],
25+
"source": [
26+
"!pip install agent-diff langchain langchain-openai tqdm pandas -q"
27+
]
28+
},
29+
{
30+
"cell_type": "code",
31+
"execution_count": null,
32+
"metadata": {},
33+
"outputs": [],
34+
"source": [
35+
"import os\n",
36+
"from getpass import getpass\n",
37+
"\n",
38+
"if not os.environ.get(\"AGENT_DIFF_API_KEY\"):\n",
39+
" os.environ[\"AGENT_DIFF_API_KEY\"] = getpass(\"Agent-Diff API key: \")\n",
40+
"\n",
41+
"if not os.environ.get(\"AGENT_DIFF_BASE_URL\"):\n",
42+
" os.environ[\"AGENT_DIFF_BASE_URL\"] = \"https://api.agentdiff.dev\"\n",
43+
"\n",
44+
"OPENROUTER_API_KEY = os.environ.get(\"OPENROUTER_API_KEY\") or getpass(\"OpenRouter API key: \")\n",
45+
"\n",
46+
"# --- Settings ---\n",
47+
"MODEL = \"deepseek/deepseek-chat-v3-0324\" # change to any OpenRouter model\n",
48+
"MAX_ITERATIONS = 40 # max agent loop turns per task\n",
49+
"MAX_TESTS = None # None = run all tests; set to e.g. 5 for a quick trial\n",
50+
"TIMEOUT_SECONDS = 480 # per-test timeout"
51+
]
52+
},
53+
{
54+
"cell_type": "code",
55+
"execution_count": null,
56+
"metadata": {},
57+
"outputs": [],
58+
"source": [
59+
"SERVICE_CONFIG = {\n",
60+
" \"slack\": {\n",
61+
" \"name\": \"Slack\",\n",
62+
" \"base_url\": \"https://slack.com/api\",\n",
63+
" \"description\": \"Slack workspace messaging and collaboration API\",\n",
64+
" \"extra_context\": \"\",\n",
65+
" \"test_suite_name\": \"Slack Bench v2\",\n",
66+
" },\n",
67+
" \"box\": {\n",
68+
" \"name\": \"Box\",\n",
69+
" \"base_url\": \"https://api.box.com/2.0\",\n",
70+
" \"description\": \"Box cloud storage and file management API\",\n",
71+
" \"extra_context\": \"\",\n",
72+
" \"test_suite_name\": \"Box Bench v2\",\n",
73+
" },\n",
74+
" \"calendar\": {\n",
75+
" \"name\": \"Google Calendar\",\n",
76+
" \"base_url\": \"https://www.googleapis.com/calendar/v3\",\n",
77+
" \"description\": \"Google Calendar scheduling and events API\",\n",
78+
" \"extra_context\": \"Current Date/Time: Sunday, June 17, 2018 at 00:01 (midnight), timezone America/Los_Angeles. Use this as the reference point for all relative date/time expressions like 'today', 'tomorrow', 'this Saturday', etc.\",\n",
79+
" \"test_suite_name\": \"Calendar Bench\",\n",
80+
" },\n",
81+
" \"linear\": {\n",
82+
" \"name\": \"Linear\",\n",
83+
" \"base_url\": \"https://api.linear.app/graphql\",\n",
84+
" \"description\": \"Linear project management and issue tracking API\",\n",
85+
" \"extra_context\": \"\",\n",
86+
" \"test_suite_name\": \"Linear Bench\",\n",
87+
" },\n",
88+
"}\n",
89+
"\n",
90+
"SYSTEM_PROMPT_TEMPLATE = \"\"\"You are an AI assistant that completes tasks by interacting with APIs via bash commands.\n",
91+
"\n",
92+
"Current Session:\n",
93+
"- Service: {service_name}\n",
94+
"- Base URL: {base_url}\n",
95+
"- Description: {service_description}\n",
96+
"{extra_context}\n",
97+
"\n",
98+
"Environment:\n",
99+
"- You are authenticated as a user in the {service_name} workspace/account.\n",
100+
"- Authentication is handled automatically via proxy. Use placeholder tokens like <TOKEN> where credentials would go.\n",
101+
"- Use the execute_bash tool to run bash commands (primarily curl) to interact with the {service_name} API.\n",
102+
"- If you are not sure how to use the {service_name} API, explore the endpoint, parameters, and learn how it works.\n",
103+
"- Parse API responses carefully - extract IDs and data needed for subsequent calls.\n",
104+
"- If a command fails, analyze the error and try a different approach.\n",
105+
"- Only declare completion when the task is fully completed (not just when you've gathered information).\n",
106+
"\"\"\"\n",
107+
"\n",
108+
"\n",
109+
"def build_system_prompt(service: str) -> str:\n",
110+
" config = SERVICE_CONFIG[service]\n",
111+
" return SYSTEM_PROMPT_TEMPLATE.format(\n",
112+
" service_name=config[\"name\"],\n",
113+
" base_url=config[\"base_url\"],\n",
114+
" service_description=config[\"description\"],\n",
115+
" extra_context=config[\"extra_context\"],\n",
116+
" )"
117+
]
118+
},
119+
{
120+
"cell_type": "code",
121+
"execution_count": null,
122+
"metadata": {},
123+
"outputs": [],
124+
"source": [
125+
"import time\n",
126+
"from langchain_openai import ChatOpenAI\n",
127+
"from langchain.agents import AgentExecutor, create_tool_calling_agent\n",
128+
"from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder\n",
129+
"from agent_diff import AgentDiff, BashExecutorProxy, create_langchain_tool\n",
130+
"\n",
131+
"\n",
132+
"def create_agent(service: str, bash_executor: BashExecutorProxy, model: str) -> AgentExecutor:\n",
133+
" \"\"\"Create a LangChain agent with the bash tool for a given service.\"\"\"\n",
134+
" llm = ChatOpenAI(\n",
135+
" base_url=\"https://openrouter.ai/api/v1\",\n",
136+
" api_key=OPENROUTER_API_KEY,\n",
137+
" model=model,\n",
138+
" temperature=0,\n",
139+
" )\n",
140+
" tool = create_langchain_tool(bash_executor)\n",
141+
" system_prompt = build_system_prompt(service)\n",
142+
"\n",
143+
" prompt = ChatPromptTemplate.from_messages([\n",
144+
" (\"system\", system_prompt),\n",
145+
" (\"human\", \"{input}\"),\n",
146+
" MessagesPlaceholder(variable_name=\"agent_scratchpad\"),\n",
147+
" ])\n",
148+
"\n",
149+
" agent = create_tool_calling_agent(llm, [tool], prompt)\n",
150+
" return AgentExecutor(\n",
151+
" agent=agent,\n",
152+
" tools=[tool],\n",
153+
" max_iterations=MAX_ITERATIONS,\n",
154+
" handle_parsing_errors=True,\n",
155+
" verbose=False,\n",
156+
" )"
157+
]
158+
},
159+
{
160+
"cell_type": "code",
161+
"execution_count": null,
162+
"metadata": {},
163+
"outputs": [],
164+
"source": [
165+
"from tqdm.auto import tqdm\n",
166+
"\n",
167+
"\n",
168+
"def run_single_test(client: AgentDiff, model: str, test, service: str) -> dict:\n",
169+
" \"\"\"Run one test: init env -> LangChain agent -> evaluate -> cleanup.\"\"\"\n",
170+
" env = None\n",
171+
" try:\n",
172+
" env = client.init_env(testId=test.id)\n",
173+
" run = client.start_run(envId=env.environmentId, testId=test.id)\n",
174+
" bash_executor = BashExecutorProxy(env.environmentId, base_url=client.base_url, api_key=client.api_key)\n",
175+
"\n",
176+
" agent_executor = create_agent(service, bash_executor, model)\n",
177+
"\n",
178+
" start = time.perf_counter()\n",
179+
" agent_output = agent_executor.invoke({\"input\": test.prompt})\n",
180+
" elapsed = time.perf_counter() - start\n",
181+
"\n",
182+
" client.evaluate_run(runId=run.runId)\n",
183+
" result = client.get_results_for_run(runId=run.runId)\n",
184+
" client.delete_env(envId=env.environmentId)\n",
185+
"\n",
186+
" return {\n",
187+
" \"test_id\": str(test.id),\n",
188+
" \"test_name\": getattr(test, \"name\", \"\"),\n",
189+
" \"passed\": result.passed,\n",
190+
" \"score\": result.score.get(\"percent\", 0) if isinstance(result.score, dict) else 0,\n",
191+
" \"failures\": result.failures,\n",
192+
" \"time\": round(elapsed, 2),\n",
193+
" \"agent_output\": agent_output.get(\"output\", \"\"),\n",
194+
" }\n",
195+
" except Exception as e:\n",
196+
" if env:\n",
197+
" try:\n",
198+
" client.delete_env(envId=env.environmentId)\n",
199+
" except Exception:\n",
200+
" pass\n",
201+
" return {\"test_id\": str(test.id), \"test_name\": getattr(test, \"name\", \"\"), \"passed\": False, \"score\": 0, \"error\": str(e)}\n",
202+
"\n",
203+
"\n",
204+
"def run_benchmark(model: str, services: list[str] | None = None, max_tests: int | None = None) -> list[dict]:\n",
205+
" \"\"\"Run the full benchmark across services using LangChain agent.\"\"\"\n",
206+
" services = services or list(SERVICE_CONFIG.keys())\n",
207+
" client = AgentDiff()\n",
208+
" all_results = []\n",
209+
"\n",
210+
" for service in services:\n",
211+
" config = SERVICE_CONFIG[service]\n",
212+
"\n",
213+
" suite_list = client.list_test_suites(name=config[\"test_suite_name\"])\n",
214+
" if not suite_list.testSuites:\n",
215+
" print(f\"[SKIP] Test suite '{config['test_suite_name']}' not found.\")\n",
216+
" continue\n",
217+
" suite = client.get_test_suite(suite_list.testSuites[0].id, expand=True)\n",
218+
" tests = suite.tests[:max_tests] if max_tests else suite.tests\n",
219+
"\n",
220+
" print(f\"\\n{'='*60}\")\n",
221+
" print(f\" {config['name']} — {len(tests)} tests | model: {model}\")\n",
222+
" print(f\"{'='*60}\")\n",
223+
"\n",
224+
" for test in tqdm(tests, desc=config[\"name\"]):\n",
225+
" result = run_single_test(client, model, test, service)\n",
226+
" result[\"service\"] = service\n",
227+
" result[\"model\"] = model\n",
228+
" all_results.append(result)\n",
229+
"\n",
230+
" status = \"PASS\" if result.get(\"passed\") else \"FAIL\"\n",
231+
" score = result.get(\"score\", 0)\n",
232+
" tqdm.write(f\" [{status}] {result.get('test_name', result['test_id'])[:60]} score={score}\")\n",
233+
"\n",
234+
" return all_results"
235+
]
236+
},
237+
{
238+
"cell_type": "code",
239+
"execution_count": null,
240+
"metadata": {},
241+
"outputs": [],
242+
"source": [
243+
"results = run_benchmark(\n",
244+
" model=MODEL,\n",
245+
" services=None, # all 4 services; or e.g. [\"slack\", \"box\"]\n",
246+
" max_tests=MAX_TESTS,\n",
247+
")"
248+
]
249+
},
250+
{
251+
"cell_type": "code",
252+
"execution_count": null,
253+
"metadata": {},
254+
"outputs": [],
255+
"source": [
256+
"import pandas as pd\n",
257+
"\n",
258+
"df = pd.DataFrame(results)\n",
259+
"\n",
260+
"print(\"\\n\" + \"=\" * 60)\n",
261+
"print(f\" Results: {MODEL} (LangChain Agent)\")\n",
262+
"print(\"=\" * 60)\n",
263+
"\n",
264+
"if \"service\" in df.columns and \"score\" in df.columns:\n",
265+
" summary = df.groupby(\"service\").agg(\n",
266+
" tests=(\"score\", \"count\"),\n",
267+
" passed=(\"passed\", \"sum\"),\n",
268+
" mean_score=(\"score\", \"mean\"),\n",
269+
" pass_rate=(\"passed\", \"mean\"),\n",
270+
" ).round(2)\n",
271+
" summary[\"pass_rate\"] = (summary[\"pass_rate\"] * 100).round(1)\n",
272+
" print(\"\\nPer-service summary:\")\n",
273+
" print(summary.to_string())\n",
274+
"\n",
275+
" overall_score = df[\"score\"].mean()\n",
276+
" overall_pass = df[\"passed\"].mean() * 100\n",
277+
" print(f\"\\nOverall: score={overall_score:.1f} pass_rate={overall_pass:.1f}%\")\n",
278+
"\n",
279+
" summary[\"mean_score\"].plot.bar(title=f\"Agent-Diff Score by Service ({MODEL}, LangChain)\", ylabel=\"Score\", xlabel=\"Service\", rot=0)\n",
280+
"else:\n",
281+
" print(df)"
282+
]
283+
}
284+
],
285+
"metadata": {
286+
"kernelspec": {
287+
"display_name": "Python 3",
288+
"language": "python",
289+
"name": "python3"
290+
},
291+
"language_info": {
292+
"name": "python",
293+
"version": "3.11.0"
294+
}
295+
},
296+
"nbformat": 4,
297+
"nbformat_minor": 4
298+
}

0 commit comments

Comments
 (0)