Add Exgentic Benchmarks MCP Server#178

Draft

yoavkatz wants to merge 16 commits intokagenti:mainfrom

yoavkatz:feature/exgentic-mcp-server

yoavkatz commented Mar 16, 2026 •

edited

Loading

Summary

This PR adds a Docker-based MCP server for Exgentic benchmarks.

For: kagenti/kagenti#963

as part of Epic: kagenti/kagenti#962

Changes

Created directory with:
- Dockerfile: Multi-stage build using uv for fast package installation
- entrypoint.sh: Runtime script to start the MCP server
- README.md: Comprehensive documentation with examples
- .dockerignore: Build optimization

Features

✅ Supports building images with specific benchmarks (e.g., tau2, webarena, miniwob)
✅ Uses uv for fast package installation
✅ Includes git configuration to handle large repository clones (HTTP/1.1, increased buffers)
✅ Benchmark is installed at build time for faster startup
✅ Configurable HOST/PORT via environment variables
✅ Runs as non-root user (UID 1001)
✅ Successfully tested with tau2 benchmark (114 tasks loaded)

Testing

Built and tested the Docker image:

Image size: 2.43 GB
Successfully loaded tau2 benchmark with 114 tasks
MCP server started and responded correctly to HTTP requests
3 management tools available (list_tasks, create_session, delete_session)

Usage

# Build for tau2 benchmark
cd mcp/exgentic_benchmarks
podman build --build-arg BENCHMARK_NAME=tau2 -t exgentic-mcp-tau2 .

# Run the server
podman run -p 8000:8000 exgentic-mcp-tau2

Documentation

See mcp/exgentic_benchmarks/README.md for complete documentation including:

Quick start guide
Build instructions for different benchmarks
Configuration options
Troubleshooting guide
Advanced usage examples


          Add Exgentic Benchmarks MCP server

c49850d

- Created Docker-based MCP server for Exgentic benchmarks
- Supports building images with specific benchmarks (e.g., tau2)
- Uses uv for fast package installation
- Includes git configuration to handle large repository clones
- Benchmark is installed at build time for faster startup
- Configurable HOST/PORT via environment variables
- Runs as non-root user (UID 1001)
- Comprehensive documentation with examples
- Successfully tested with tau2 benchmark (114 tasks loaded)

Signed-off-by: Yoav Katz <katz@il.ibm.com>

yoavkatz requested a review from a team as a code owner

March 16, 2026 14:22

yoavkatz marked this pull request as draft

March 16, 2026 14:23

pdettori requested a review from kellyaa

March 16, 2026 14:49

yoavkatz added 3 commits

March 17, 2026 13:48


          Add runtime configuration support via --set parameters

b8bda2f

- Modified entrypoint.sh to support EXGENTIC_SET_* environment variables
- Environment variables are converted to --set arguments for exgentic mcp command
- Format: EXGENTIC_SET_BENCHMARK_USER_SIMULATOR_MODEL -> --set benchmark.user_simulator_model
- Updated README with detailed documentation and examples
- Added common parameter list and usage examples
- Supports setting user_simulator_model, agent_model, max_steps, and other benchmark parameters

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Added benchmark name for default .env examples

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Fixed env names in .env.openai

a03890f

Signed-off-by: Yoav Katz <katz@il.ibm.com>

esnible reviewed

View reviewed changes

Contributor

esnible left a comment

Can you add, either to the README, or to this PR, the rationale for adding this to the Kagenti examples? Have you been testing this with Kagenti -- the README only talks about testing with Docker. The instructions talk about different builds but Kagenti builds only once -- do we need those instructions?

mcp/exgentic_benchmarks/README.md Outdated

yoavkatz added 2 commits

March 17, 2026 17:41


          Fix Kubernetes compatibility and DNS rebinding protection

f9af7a8

- Updated Dockerfile permissions for Kubernetes compatibility (group permissions)
- Changed ownership from 1001:1001 to 1001:0 for OpenShift/Kubernetes
- Added group read/write/execute permissions (g+rwX)
- Added --disable-dns-rebinding-protection flag to allow Kubernetes service access
- This fixes the 'Invalid Host header' error when accessing from within cluster

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Added link to Exgentic , per review comment

6b54e97

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Author

yoavkatz commented Mar 18, 2026

Can you add, either to the README, or to this PR, the rationale for adding this to the Kagenti examples? Have you been testing this with Kagenti -- the README only talks about testing with Docker. The instructions talk about different builds but Kagenti builds only once -- do we need those instructions?

This PR is related to work. It extends the work done by @kellyaa to add appworld mcp server to encompass in a systematic way multiple benchmarks supported by the Exgentic framework.

I added now the references to the related Kagenti issue (kagenti/kagenti#963) and Epic (kagenti/kagenti#962) for clearer context.

yoavkatz added 2 commits

March 18, 2026 12:44


          Fixed model gpt4o name

c86118e

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          fix: Set HOME=/app in exgentic benchmarks Dockerfile

fef6cb0

- Added ENV HOME=/app after USER 1001 to ensure proper home directory
- Deleted .env.openai (replaced with benchmark-specific env files)
- This ensures tools/libraries write to /app where user has permissions

Signed-off-by: Yoav Katz <katz@il.ibm.com>

rubambiza mentioned this pull request

Org Weekly Report 2026-03-16 -- 2026-03-23 kagenti/kagenti#1094

Closed

rubambiza mentioned this pull request

Org Weekly Report 2026-03-23 -- 2026-03-30 kagenti/kagenti#1110

Closed

yoavkatz added 8 commits

March 31, 2026 21:25


          Add git-lfs installation to exgentic_benchmarks Dockerfile

470af46

- Install git-lfs package alongside git
- Initialize git-lfs after installation
- Update comment to reflect git-lfs requirement

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Add build.sh script for exgentic_benchmarks

3981b2a


          Updated to use latest version of exgentic code

38fa433

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Add Exgentic A2A agent Docker wrapper

cb4fe83

- Add Dockerfile for building Exgentic agents with A2A protocol support
- Add entrypoint.sh for runtime configuration and MCP integration
- Add build.sh script with auto-detection of docker/podman
- Add comprehensive README with usage examples and configuration
- Add .dockerignore for optimized build context
- Add .env.example and .env.advanced for configuration templates
- Support runtime configuration via EXGENTIC_SET_* environment variables
- Implement MCP server integration via --mcp flag
- Run as non-root user (UID 1001) for security

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Refactor build scripts to use named arguments and --use-cache flag

948abf6

- Change build scripts to accept agent/benchmark as positional argument
- Replace --no-cache with --use-cache flag (builds without cache by default)
- Update argument parsing to support --tag and --use-cache as optional flags
- Improve help messages with clearer usage examples
- Update README files to reflect new command syntax
- Default behavior: builds without cache for consistency
- Optional: use --use-cache to enable Docker cache during builds

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Set example env vars.

ceccd3a

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          Fix benchmark data installation path in Dockerfile

80ea722

- Set HOME=/app before running exgentic install during build
- This ensures benchmark data is installed to /app/.exgentic/ instead of /root/.exgentic/
- Fixes runtime error where benchmark data files were not found
- Remove duplicate HOME environment variable declaration

Signed-off-by: Yoav Katz <katz@il.ibm.com>


          fix(exgentic-benchmarks): resolve HOME directory issue for Kubernetes…

47434e5

… deployment

- Changed user UID from 1001 to 1000 to match Kubernetes securityContext
- Set home directory to /app during user creation with useradd -d flag
- Added explicit HOME=/app environment variable after USER directive
- Added HOME export in entrypoint.sh to ensure propagation to child processes
- Added TAU2_DATA_DIR environment variable for benchmark data location

This fixes the 'RuntimeError: Could not determine home directory' error
that occurred when Exgentic spawned child processes in Kubernetes. The
issue was caused by a mismatch between the Kubernetes runAsUser (1000)
and the Dockerfile user UID (1001), resulting in no /etc/passwd entry
for the running process, which prevented Python's Path.expanduser()
from resolving the home directory in child processes.

Signed-off-by: Yoav Katz <katz@il.ibm.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet