diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index 732ac25f..4d95c2a5 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -1,9 +1,101 @@ --- -title: Measure and improve +title: Measure and improve a factory description: >- - Factory measurement and improvement documentation will be added in a follow-up PR. + Measure factory activity and costs, evaluate completed conversations, compare + agent configurations, and turn failures into follow-up work. sidebar: label: "Measure and improve" +topic: factories --- -Factory measurement and improvement documentation will land in a follow-up PR. +Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it. + +| Feature | What it tells you | +| --- | --- | +| Dashboard metrics | How much work the factory produced, and what it cost. | +| Scorers | Whether completed conversations meet criteria you define. | +| Benchmarks | How different configurations perform on the same tasks. | +| Self-improvement | Which repeated failures get investigated and turned into follow-up work. | + +## Read dashboard metrics + +The control room dashboard shows activity, cost, autonomy, and evaluation results: + +| Metric | What it shows | +| --- | --- | +| **Total runs** | All agent runs, with breakdowns by agent type, status, source, model, and more. | +| **PRs opened** | Pull requests created from factory work. | +| **PRs merged** | Pull requests that merged. | +| **Autonomy %** | The share of merged work that shipped without human edits. | +| **Time saved** | An estimate of engineer-hours saved by merged factory PRs. | +| **PR latency** | How long pull requests spend in each stage. | +| **Cost per PR** | An estimate of credits spent per pull request. | +| **Most expensive PRs** | The highest-cost pull requests. | +| **Scorer cards** | Results from your Scorers. | +| **Self-improvement PRs** | The three newest Self-improvement pull requests, regardless of the selected date range. | + +**Time saved** and **Cost per PR** are estimates, not billing figures. Time saved converts merged pull requests into engineer-hours, and the per-teammate view divides that estimate by your team size. Cost per PR counts recorded credits and can undercount actual usage. + +:::caution +**PRs merged**, **PR latency**, and the detail in **Most expensive PRs** require the GitHub App and only cover activity from after you install it. The ratio of **PRs merged** to **PRs opened** isn't a merge rate; the two are counted differently. Run counts include evaluation, benchmark, and Self-improvement runs. +::: + +Use the dashboard to pick which runs to investigate, not to conclude what caused a change. A higher run count with a flat PR count could mean harder tasks, retries, or measurement activity. + +## Configure Scorers + +A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as "did the agent run the tests before opening a PR?" Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. + +Configure these fields: + +* **Judge instructions** - The criteria the judge checks for. +* **Classifications** - The labels the judge can assign, each with a score. +* **Pass threshold** - The score a conversation needs to pass. +* **Sample rate** - The portion of completed conversations to evaluate. +* **Judge model** - The model that acts as the judge. + +You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after a conversation completes, the judge evaluates it and records a classification, a score, and its reasoning. + +| Mode | Use it when | You get | +| --- | --- | --- | +| Manual | You want to score one conversation or test new judge instructions. | An evaluation of the selected conversation. | +| Periodic | You want an ongoing sample to track over time. | A baseline to compare against after a change. | + +Changing **Pass threshold** updates how past scores display as pass or fail; the recorded results don't change. + +## Compare configurations with benchmarks + +A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A suite includes: + +* **Agent** - The agent whose configurations you compare. +* **Tasks** - Fixed prompts with success criteria. +* **Configurations** - The harness, model, and runner combinations to test. +* **Scorers** - Your classification Scorers, applied to every trial. +* **Repetitions** - The number of trials per task and configuration. + +You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. + +Every benchmark also runs **Correctness**, a built-in scorer that marks each trial as pass or fail against the task's success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don't include model usage, so the true cost is higher. + +## Configure and review Self-improvement + +Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code, or to the factory's prompts, skills, and configuration when that repo is available. Follow-up runs can open pull requests, but nothing is adopted without your review. + +The factory-level **Analysis model** setting chooses the model used for this analysis. Factories managed from files in a GitHub repo can't change this setting in the control room. + +Each Self-improvement pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence. + +## Run a practical improvement loop + +Change one measurable thing at a time: + +1. **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then compare the judge's results against a few conversations you review yourself. +2. **Collect a baseline.** Let periodic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs. +3. **Inspect failures.** Read the judge's reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating. +4. **Benchmark a candidate.** Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference. +5. **Review and adopt.** If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones. +6. **Keep monitoring.** Leave the Scorer active and compare new results against your baseline. Pause or revise the Scorer when its criteria no longer match what your team needs. + +## Next step + +Record an adopted change in [factory definitions as code](./factory-as-code) so your team can review the factory configuration.