Redesign VeRO as a generic program optimization harness

.gitignore

@@ -1,5 +1,6 @@
 # Python-generated files
 __pycache__/
+.pytest_cache/
 *.py[oc]
 build/
 dist/

README.md

@@ -1,122 +1,72 @@
-# VeRO: Versioning Rewards and Observations
+# VeRO: a harness for agents to optimize programs
 
 [![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 
-VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.
+VeRO gives a coding agent a program to edit, an evaluation boundary, and durable
+memory of every candidate it tried. The target can be an agent, a prompt, a
+compiler pass, a CUDA kernel, a matrix multiplication function, or any other
+Git-versioned program.
 
-> **Paper**: [VeRO: An Evaluation Harness for Agents to Optimize Agents](https://arxiv.org/abs/2602.22480)
-
-## Repository Structure
-
-```
-vero/
-├── vero/               # Core library (scale-vero)
-├── vero-agents/        # Agent implementations (benchmarking targets)
-├── vero-benchmarking/  # Benchmarking scripts and analysis
-└── LICENSE
-```
-
-### [vero/](vero/)
-
-The core optimization framework. Provides:
-
-- **Policy** — orchestrates the optimization loop (agent + evaluator + git)
-- **Agents** — VeroAgent (OpenAI Agents SDK) and ClaudeCodeAgent (Claude Agent SDK)
-- **Evaluator** — runs task evaluations in isolated subprocess environments
-- **Tools** — MCP-based tools for agents (bash, file I/O, experiment runner, dataset viewer, etc.)
-- **Traces** — session analysis and LLM-based trace interpretation
-
-```bash
-cd vero && uv sync --extra optimize
+```text
+strategy -> candidate producers -> evaluation backends -> selection
+                  |                       |
+          isolated workspaces       versioned reports
 ```
 
-See [vero/README.md](vero/README.md) for full documentation.
-
-### [vero-agents/](vero-agents/)
+The target and evaluator do not need to be Python. External evaluators and
+candidate producers connect through command protocols; Python benchmarks can
+use the optional, optimizer-independent `scale-vero-tasks` package.
 
-Agent implementations used as optimization targets:
+Targets may live locally or in an isolated sandbox. VeRO keeps optimization
+state and experiment tracking on the host while running Git worktrees, producer
+commands, builds, and evaluation commands in the target sandbox. The core guide
+includes a no-bind-mount `DockerSandbox` example.
 
-| Agent | Description |
-|-------|-------------|
-| **generic-agent** | General-purpose agent for MATH, GPQA, GAIA, GSM8K, etc. |
-| **web_search_agent** | Web search agent for SimpleQA, Facts Search |
-| **KIRA** | Terminal task agent for Terminal Bench 2.0 |
-| **tau-bench** | Customer service tool-use agent |
-| **pharma_summarizer** | Document summarization agent |
+## Repository layout
 
-See [vero-agents/README.md](vero-agents/README.md) for details.
+| Directory | Purpose |
+| --- | --- |
+| [`vero/`](vero/) | The `scale-vero` optimization kernel, runtime, CLI, and coding-agent adapters |
+| [`vero-tasks/`](vero-tasks/) | Narrow Python task types and schema-v1 evaluation runner |
+| [`vero-agents/`](vero-agents/) | Benchmark target programs; not part of the optimizer runtime |
+| [`vero-benchmarking/`](vero-benchmarking/) | Reproducible experiment configurations and drivers |
 
-### [vero-benchmarking/](vero-benchmarking/)
-
-Scripts and infrastructure for running optimization experiments:
+Start with the [generic C matrix-multiplication quickstart](vero/examples/c-matmul/)
+or the [core guide](vero/README.md). The C target has no Python or VeRO
+dependency; it is compiled, checked, benchmarked, and optimized through the
+language-neutral command protocol.
 
 ```bash
-cd vero-benchmarking && uv sync --all-extras
-
-# Run an optimization experiment
-uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math
-
-# Build datasets
-./scripts/build_datasets.sh
+cd vero
+uv sync --all-extras
+uv run vero --help
 ```
 
-See [vero-benchmarking/README.md](vero-benchmarking/README.md) for full documentation.
-
-## Quick Start
-
-### Prerequisites
+## Paper reproduction
 
-- Python 3.11+
-- [uv](https://docs.astral.sh/uv/getting-started/installation/)
-- Git
-- Access to an LLM provider (via LiteLLM, OpenAI, Anthropic, etc.)
+VeRO was introduced in [*VeRO: A Harness for Agents to Optimize
+Agents*](https://arxiv.org/abs/2602.22480), accepted at ICML 2026. The current
+library generalizes that version/evaluate/select loop from agents to programs.
 
-### Install
+The exact paper implementation is frozen separately from the v0.5 redesign:
 
 ```bash
-git clone <repo-url> && cd vero
-
-# Install core library
-cd vero && uv sync --extra optimize
-
-# Install benchmarking tools
-cd ../vero-benchmarking && uv sync --all-extras
+git checkout paper-v1
 ```
 
-### Run Your First Optimization
-
-```python
-from agents import Agent as OAIAgent
-from vero.policy import Policy
-from vero.agents.vero import VeroAgent
-
-policy = Policy(
-    project_path="/path/to/my-agent",
-    dataset="/path/to/my-dataset",
-    agent=VeroAgent(
-        oai_agent=OAIAgent(name="VeroAgent", model="anthropic/claude-sonnet-4-5-20250929"),
-    ),
-    task="main",
-    train_budget=10,
-    max_turns=200,
-)
-
-best = await policy.run()
-print(f"Best commit: {best.commit}, score: {best.score}")
-```
+Use the `paper/v1` branch or `paper-v1` tag for reproduction. Development of
+the generic program optimizer continues on the `v0.5` branch.
 
 ## Citation
 
 ```bibtex
 @article{ursekar2026vero,
-  title={VeRO: An Evaluation Harness for Agents to Optimize Agents},
+  title={VeRO: A Harness for Agents to Optimize Agents},
   author={Ursekar, Varun and Shanker, Apaar and Chatrath, Veronica and Xue, Yuan (Emily) and Denton, Sam},
   journal={arXiv preprint arXiv:2602.22480},
   year={2026}
 }
 ```
 
-## License
-
-[MIT](LICENSE)
+VeRO is licensed under the [MIT License](LICENSE).

vero-agents/README.md

@@ -1,6 +1,12 @@
-# Vero Agents
+# VeRO benchmark targets
 
-A collection of Python-based agent implementations optimizable with [VeRO](../vero/). Each agent is a self-contained `uv` package with its own dependencies and evaluation tasks.
+This directory is VeRO's benchmark corpus: Python programs used as optimization
+targets in the paper and in end-to-end tests. Agents are one important kind of
+target program, but they are not part of the optimizer runtime.
+
+Each target is a self-contained uv project. It depends on the narrow
+[`scale-vero-tasks`](../vero-tasks/) protocol for Python evaluation ergonomics,
+not on `scale-vero` itself.
 
 ## Agents
 
@@ -25,24 +31,26 @@ vero-agents/
 └── pyproject.toml
 ```
 
-Each agent contains:
-- `pyproject.toml` with dependencies and a dev dependency on `scale-vero`
+Each target contains:
+
+- `pyproject.toml` with its own runtime dependencies and `scale-vero-tasks`
 - Source code under `src/<agent_name>/` or `<agent_name>/`
-- A `vero_tasks/` module defining evaluation tasks for VeRO
+- A `vero_tasks/` module defining inference and scoring functions
 
-## Adding a New Agent
+## Adding a new target
 
 1. Create a directory under `agents/`
 2. Initialize a `uv` package with `pyproject.toml`
-3. Add `scale-vero` as a dev dependency:
+3. Add the task protocol:
+
    ```toml
-   [dependency-groups]
-   dev = ["scale-vero[evaluate]"]
+   [project]
+   dependencies = ["scale-vero-tasks>=0.1.0"]
 
    [tool.uv.sources]
-   scale-vero = { path = "../../../vero", editable = true }
+   scale-vero-tasks = { path = "../../../vero-tasks", editable = true }
    ```
-4. Create a `vero_tasks/` module with inference and evaluation functions (see `vero init tasks`)
+4. Create a `vero_tasks/` module with inference and evaluation functions
 5. Register the task in `vero-benchmarking/src/vero_benchmarking/tasks/`
 
 ## Running Evaluations

vero-agents/agents/KIRA/pyproject.toml

@@ -9,13 +9,9 @@ dependencies = [
     "datasets",
     "harbor",
     "litellm",
+    "scale-vero-tasks>=0.1.0",
     "tenacity",
 ]
 
-[dependency-groups]
-dev = [
-    "scale-vero[evaluate]",
-]
-
 [tool.uv.sources]
-scale-vero = { path = "../../../vero", editable = true }
+scale-vero-tasks = { path = "../../../vero-tasks", editable = true }

vero-agents/agents/KIRA/terminus_kira/vero_tasks/terminal_bench.py

@@ -10,9 +10,7 @@
 from __future__ import annotations
 
 from harbor import Job
-from vero.core.db.result import TaskResult
-from vero.core.evaluation import EvaluationParameters
-from vero.core.task import create_task
+from vero_tasks import TaskContext, TaskResult, create_task
 
 from terminus_kira.vero_tasks.utils import (
     TerminalBenchTask,
@@ -25,7 +23,7 @@
 
 
 @terminal_bench_2("run_inference")
-async def run_inference(task: TerminalBenchTask, evaluation_parameters: EvaluationParameters):
+async def run_inference(task: TerminalBenchTask, evaluation_parameters: TaskContext):
     """No-op — Harbor handles inference and evaluation together."""
     return None
 
@@ -34,7 +32,7 @@ async def run_inference(task: TerminalBenchTask, evaluation_parameters: Evaluati
 async def run_evaluation(
     tasks: list[TerminalBenchTask],
     outputs: list[None],
-    evaluation_parameters: EvaluationParameters,
+    evaluation_parameters: TaskContext,
 ) -> list[TaskResult]:
     """Run a Harbor Job for a batch of Terminal-Bench tasks.
 

vero-agents/agents/KIRA/terminus_kira/vero_tasks/utils.py

@@ -17,8 +17,7 @@
 from harbor.models.trial.config import AgentConfig, EnvironmentConfig
 from harbor.models.trial.result import TrialResult
 from pydantic import Field
-from vero.core.db.result import TaskResult
-from vero.core.evaluation import EvaluationParameters, TaskParameters
+from vero_tasks import TaskContext, TaskParameters, TaskResult
 
 
 def load_trial_results(job_dir: Path) -> list[TrialResult]:
@@ -65,7 +64,7 @@ class KiraParameters(TaskParameters):
 
 def build_job_config(
     tasks: list[TerminalBenchTask],
-    evaluation_parameters: EvaluationParameters,
+    evaluation_parameters: TaskContext,
 ) -> JobConfig:
     """Build a Harbor JobConfig from vero evaluation parameters.
 

vero-agents/agents/generic-agent/pyproject.toml

@@ -9,6 +9,7 @@ authors = [
 requires-python = ">=3.11"
 dependencies = [
     "openai-agents[litellm]>=0.10",
+    "scale-vero-tasks>=0.1.0",
     "sympy>=1.14.0",
     "tree-sitter>=0.25.2",
 ]
@@ -25,8 +26,7 @@ dev = [
     "pytest>=9.0.2",
     "pytest-asyncio>=1.3.0",
     "pytest-json-report>=1.5.0",
-    "scale-vero[evaluate]",
 ]
 
 [tool.uv.sources]
-scale-vero = { path = "../../../vero", editable = true }
+scale-vero-tasks = { path = "../../../vero-tasks", editable = true }

vero-agents/agents/generic-agent/src/generic_agent/prompts.py

@@ -1,35 +1,30 @@
 from collections.abc import Callable
 from typing import Any
 
-from vero.core.resource import resource
 
 PromptFormatter = Callable[..., str]
 
 
-@resource(namespace="gsm8k", name="prompt")
 def get_gsm8k_prompt(question: str, **kwargs: Any) -> str:
     """Format prompt for GSM8K benchmark."""
     GSM8K_DEFAULT_PROMPT = """{question}
 Generate an answer to this question. At the end, provide the final answer in the format "Answer is <number>", where <number> is a single number."""
     return GSM8K_DEFAULT_PROMPT.format(question=question)
 
 
-@resource(namespace="math", name="prompt")
 def get_math_prompt(question: str, **kwargs: Any) -> str:
     """Format prompt for MATH benchmark."""
     MATH_DEFAULT_PROMPT = """{question}
 Please generate a solution for the problem. At the end, provide the final answer in the format "\\boxed{{<number>}}", where <number> is a math answer(an expression or number), without any additional information or explanation."""
     return MATH_DEFAULT_PROMPT.format(question=question)
 
 
-@resource(namespace="hotpotqa", name="prompt")
 def get_hotpotqa_prompt(question: str, context: str, **kwargs: Any) -> str:
     """Format prompt for HotpotQA benchmark."""
     HOTPOTQA_DEFAULT_PROMPT = "Context: {context}\n\nQuestion: {question}\n\nAnswer:"
     return HOTPOTQA_DEFAULT_PROMPT.format(question=question, context=context)
 
 
-@resource(namespace="drop", name="prompt")
 def get_drop_prompt(question: str, passage: str, **kwargs: Any) -> str:
     """Format prompt for DROP benchmark."""
     DROP_DEFAULT_PROMPT = """Given a question and a passage, please answer the question.
@@ -40,22 +35,19 @@ def get_drop_prompt(question: str, passage: str, **kwargs: Any) -> str:
     return DROP_DEFAULT_PROMPT.format(question=question, passage=passage)
 
 
-@resource(namespace="humaneval", name="prompt")
 def get_humaneval_prompt(question: str, **kwargs: Any) -> str:
     """Format prompt for HumanEval benchmark."""
     HUMANEVAL_DEFAULT_PROMPT = """{question}
 Generate an answer to this question, without any additional test cases."""
     return HUMANEVAL_DEFAULT_PROMPT.format(question=question)
 
 
-@resource(namespace="mbpp", name="prompt")
 def get_mbpp_prompt(question: str, test_list: str, **kwargs: Any) -> str:
     """Format prompt for MBPP benchmark."""
     MBPP_DEFAULT_PROMPT = """You are an expert Python programmer, and here is your task: {question} Your code should pass these tests:\n\n{test_list}\n"""
     return MBPP_DEFAULT_PROMPT.format(question=question, test_list=test_list)
 
 
-@resource(namespace="gpqa", name="prompt")
 def get_gpqa_prompt(question: str, options: str, **kwargs: Any) -> str:
     """Format prompt for GPQA benchmark."""
     GPQA_TEMPLATE = """Answer the following multiple choice question. The last line of your response should be of the following format: ‘Answer: $LETTER’ (without quotes) where LETTER is one of ABCD. Think step by step before answering.
@@ -72,7 +64,6 @@ def get_gpqa_prompt(question: str, options: str, **kwargs: Any) -> str:
     return GPQA_TEMPLATE.format(**kwargs)
 
 
-@resource(namespace="gaia", name="prompt")
 def get_gaia_prompt(question: str, file_path: str | None = None, **kwargs: Any) -> str:
     """
     Format prompt for GAIA benchmark.
@@ -94,7 +85,6 @@ def get_gaia_prompt(question: str, file_path: str | None = None, **kwargs: Any)
     return prompt
 
 
-@resource(namespace="default", name="prompt")
 def get_default_prompt(**kwargs: Any) -> str:
     """
     Default formatting function.