From aaecb28a8130fc21c9086ed274dc11181f61747c Mon Sep 17 00:00:00 2001 From: Hong Yi Chen Date: Sat, 15 Aug 2026 16:44:39 -0700 Subject: [PATCH 1/4] docs(factories): document measurement and improvement Co-Authored-By: Warp Agent --- .../docs/factories/measure-and-improve.mdx | 104 +++++++++++++++++- 1 file changed, 101 insertions(+), 3 deletions(-) diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index 732ac25f..4aba2f8f 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -1,9 +1,107 @@ --- -title: Measure and improve +title: Measure and improve a factory description: >- - Factory measurement and improvement documentation will be added in a follow-up PR. + Measure factory activity and costs, evaluate completed conversations, compare + agent configurations, and turn failures into follow-up work. sidebar: label: "Measure and improve" +topic: factories --- -Factory measurement and improvement documentation will land in a follow-up PR. +Warp Factories records activity and agent outcomes so you can identify a failure, test a specific change, and decide whether to adopt it. + +| Capability | Question answered | When to use | Limitation | +| --- | --- | --- | --- | +| Dashboard metrics | How much work, cost, and autonomy did the factory produce? | Compare periods and investigate activity changes. | Some coverage requires the GitHub App, and measurement runs count as activity. | +| Scorers | Did completed conversations meet criteria you define? | Establish a baseline and classify failures. | Scorers are user-defined and classification-only. | +| Benchmarks | How do candidate configurations compare on fixed tasks? | Test a model, harness, or runner change. | Results do not select a winner or report full cost. | +| Self-improvement | Which repeated failures need follow-up work? | Investigate failing scores and group related causes. | It can produce pull requests, but guarantees neither a pull request nor adoption. | + +## Read dashboard metrics + +The control room surfaces activity, cost, autonomy, and evaluation results: + +| Surface | What it shows | Boundary | +| --- | --- | --- | +| **Total runs** | Agent runs, with breakdowns by agent type, status, source, root versus subruns, model, and harness. | Includes system measurement and improvement runs. | +| **PRs opened** | Pull requests associated with factory work. | Uses a different collection path from **PRs merged**. | +| **PRs merged** | Pull requests recorded as merged. | Requires the GitHub App and has no data from before collection started. | +| **Autonomy %** | Share of merged work completed without human edits. | Derived from recorded PR signals, not a quality judgment. | +| **Time saved** | A heuristic estimate from merged factory pull requests. | Uses a fixed heuristic, not billing or ROI. | +| **PR latency** | Time a pull request spends in each stage. | Stage data depends on webhook coverage and can be incomplete. | +| **Cost per PR** | A lower-bound estimate from recorded compute, platform, and inference credits. | Dollar amounts use your team's billing-tier credit rate for display only, can omit usage, and can differ from billing. | +| **Most expensive PRs** | Highest-cost pull requests, with available detail. | Detailed coverage requires the GitHub App. | +| **Scorer cards** | Results from configured Scorers. | Represents the classifications your team defined. | +| **Self-improvement PRs** | Recent pull requests produced through Self-improvement. | Shows the three newest, independent of the dashboard date range. | + +**Time saved** applies a heuristic of 25 finished lines per engineer-hour to merged factory pull requests. The per-teammate view divides that estimate by your current team size. Treat it as a directional heuristic, not a financial return or a billing figure. + +**Cost per PR** shows a median headline with the mean available, and offers **By complexity** and **By size** views. **Most expensive PRs** detail can break out inference usage by model and show a high-level human-touch or autonomy indicator. Treat that indicator as descriptive context; this page does not define additional classification rules. Dollar amounts convert recorded credits using your team's billing-tier credit rate, but remain presentational and do not represent billing. + +:::caution +Merged, latency, and detailed PR coverage is GitHub App-only, starts when collection begins, and is not backfilled. **PRs opened** and **PRs merged** use different data sources, so do not interpret their ratio as a merge rate. Run count includes evaluation, benchmark, and Self-improvement activity. Cost per PR and Time saved are estimates, not financial ROI calculations. +::: + +Use dashboard changes to choose runs for investigation, not to infer a cause. A higher run count with a stable opened-PR count might reflect harder work, retries, measurement activity, or unclear agent instructions. + +## Configure Scorers + +A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that benchmarks run automatically. Define one user-defined Scorer for one decision so its failures remain actionable. + +Configure these fields: + +* **Judge instructions** - State the observable criteria the judge applies. +* **Classifications** - Define the allowed labels and their scores. +* **Pass threshold** - Determines which recorded scores pass or fail. +* **Sample rate** - Sets the portion of eligible conversations to evaluate. +* **Judge model** - Selects a supported model or an auto router. + +You can scope a Scorer to all agents or selected agents, then keep it active or pause it. The judge evaluates eligible completed conversations after a quiet period and returns a classification, score, and reasoning. + +| Mode | Use when | Outcome | +| --- | --- | --- | +| Manual | Investigating one completed run or testing judge instructions. | An evaluation for the selected conversation. | +| Periodic | Monitoring a sample of eligible conversations over time. | A baseline for comparison before and after a change. | + +Changing **Pass threshold** reclassifies historical scores against the current threshold when you view them. It does not change the recorded classification, score, or reasoning. + +## Compare configurations with benchmarks + +A benchmark suite fixes one agent and compares configurations of it on the same tasks. Define the suite, then launch it with the configurations you want to test: + +* **Agent** - The single suite-level agent whose configurations you compare. +* **Tasks** - Fixed prompts with explicit success criteria that freeze for a benchmark launch. +* **Configurations** - Launch-time combinations of harness, model, and runner. +* **Scorers** - User-defined classification Scorers applied to every trial alongside built-in **Correctness**. +* **Repetitions** - Multiple trials per task and configuration. + +You can create a benchmark task from a completed run's detail pane. Warp copies the run's source input into the task. Define its success criteria before launch so **Correctness** evaluates every trial against the frozen task definition. + +Every benchmark automatically runs built-in **Correctness**, a Warp-managed scorer that returns a binary pass or fail against the task's frozen success criteria. Results include a per-configuration scoreboard, pass rate by Scorer, cost and quality scatter, and per-task comparison. **Correctness** and user-defined Scorer results provide separate quality signals; Warp Factories does not combine them into a numeric quality score or select a winning configuration. Benchmark credit totals cover recorded compute and platform usage, exclude inference usage, and do not represent full USD cost. + +Use enough repetitions to distinguish a stable difference from one successful trial, and apply constraints the suite does not encode, including required tools, security policy, provider availability, and the severity of each classification. + +## Configure and review Self-improvement + +Toggle **Self-improvement** on for each Scorer whose failing results warrant automated investigation. The factory-level **Analysis model** selects the model used to analyze failures and cluster related findings. GitHub-backed, file-managed factories cannot set **Analysis model** in the control room. + +Self-improvement dispatches investigation workers, clusters their findings, and files ordinary follow-up factory tasks as agent runs. A follow-up run can propose a change to application code or, when the factory configuration repo is available, to prompts, skills, or configuration. Self-improvement can produce pull requests, but it does not guarantee a pull request or automatically adopt a change. + +The dashboard's **Self-improvement PRs** shows the three newest pull requests from Self-improvement, independent of the dashboard date range. Each includes a deterministic **Regressions addressed** block that links the source runs, the failed Scorer and classification, and the triage run, so you can trace a proposed change back to the evidence that prompted it. + +## Run a practical improvement loop + +Change one measurable part of the factory at a time: + +1. **Define a Scorer.** Choose one agent role and observable failure mode. Write **Judge instructions** and **Classifications**, then set the threshold, sample rate, judge model, scope, and state. Confirm that results match several manual reviews. +2. **Collect a baseline.** Run periodic scoring until the classification distribution represents normal work. Record Scorer settings, the date range, activity metrics, and relevant cost components. +3. **Inspect failures.** Read judge reasoning and conversations, then identify causes such as missing context, unclear instructions, unavailable tools, or incomplete validation. Enable Self-improvement only when repeated failures warrant grouped investigation. +4. **Benchmark a candidate.** Compare configurations of one agent across the same tasks and repetitions. Review the configuration scoreboard, pass rate by Scorer, cost and quality scatter, per-task results, and outliers. +5. **Review and adopt.** Decide whether the evidence supports changing a harness, model, runner, parameter, prompt, skill, or other definition. Review any Self-improvement pull request or follow-up task using the same standards as a human-authored change. +6. **Continue monitoring.** Keep the Scorer active, compare results with the baseline, and watch for regressions. Pause or revise a Scorer when its rubric no longer represents the team's requirements. + +A narrow change with a stable benchmark and Scorer provides stronger evidence than a broad reconfiguration with several possible causes. + +## Next step + +Record an adopted change in [factory definitions as code](./factory-as-code) so your team can review the factory configuration. From b41d6ce4dd192e6834dbcd8ab415a78a7a0b9845 Mon Sep 17 00:00:00 2001 From: hongyi-chen Date: Sun, 16 Aug 2026 05:02:32 +0000 Subject: [PATCH 2/4] Simplify measure-and-improve language Rewrite the page in plainer language based on review feedback: - Rename the jargon table headers (Boundary/Limitation -> Keep in mind) - Drop the 25-lines-per-engineer-hour heuristic explanation; describe Time saved simply as a directional estimate - Merge the two dense metric-caveat paragraphs into one - Shorten Scorer, benchmark, Self-improvement, and improvement-loop sections without changing verified behavior claims Co-Authored-By: Warp --- .../docs/factories/measure-and-improve.mdx | 110 +++++++++--------- 1 file changed, 54 insertions(+), 56 deletions(-) diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index 4aba2f8f..fd2ddb61 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -8,99 +8,97 @@ sidebar: topic: factories --- -Warp Factories records activity and agent outcomes so you can identify a failure, test a specific change, and decide whether to adopt it. +Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it. -| Capability | Question answered | When to use | Limitation | -| --- | --- | --- | --- | -| Dashboard metrics | How much work, cost, and autonomy did the factory produce? | Compare periods and investigate activity changes. | Some coverage requires the GitHub App, and measurement runs count as activity. | -| Scorers | Did completed conversations meet criteria you define? | Establish a baseline and classify failures. | Scorers are user-defined and classification-only. | -| Benchmarks | How do candidate configurations compare on fixed tasks? | Test a model, harness, or runner change. | Results do not select a winner or report full cost. | -| Self-improvement | Which repeated failures need follow-up work? | Investigate failing scores and group related causes. | It can produce pull requests, but guarantees neither a pull request nor adoption. | +| Feature | What it tells you | Keep in mind | +| --- | --- | --- | +| Dashboard metrics | How much work the factory produced, and what it cost. | Some metrics require the GitHub App, and measurement runs count as activity. | +| Scorers | Whether completed conversations meet criteria you define. | Scorers classify conversations; they don't grade quality on a numeric scale. | +| Benchmarks | How different configurations perform on the same tasks. | Results don't pick a winner or include full cost. | +| Self-improvement | Which repeated failures need follow-up work. | It can open pull requests, but you review and adopt every change yourself. | ## Read dashboard metrics -The control room surfaces activity, cost, autonomy, and evaluation results: +The control room dashboard shows activity, cost, autonomy, and evaluation results: -| Surface | What it shows | Boundary | +| Metric | What it shows | Keep in mind | | --- | --- | --- | -| **Total runs** | Agent runs, with breakdowns by agent type, status, source, root versus subruns, model, and harness. | Includes system measurement and improvement runs. | -| **PRs opened** | Pull requests associated with factory work. | Uses a different collection path from **PRs merged**. | -| **PRs merged** | Pull requests recorded as merged. | Requires the GitHub App and has no data from before collection started. | -| **Autonomy %** | Share of merged work completed without human edits. | Derived from recorded PR signals, not a quality judgment. | -| **Time saved** | A heuristic estimate from merged factory pull requests. | Uses a fixed heuristic, not billing or ROI. | -| **PR latency** | Time a pull request spends in each stage. | Stage data depends on webhook coverage and can be incomplete. | -| **Cost per PR** | A lower-bound estimate from recorded compute, platform, and inference credits. | Dollar amounts use your team's billing-tier credit rate for display only, can omit usage, and can differ from billing. | -| **Most expensive PRs** | Highest-cost pull requests, with available detail. | Detailed coverage requires the GitHub App. | -| **Scorer cards** | Results from configured Scorers. | Represents the classifications your team defined. | -| **Self-improvement PRs** | Recent pull requests produced through Self-improvement. | Shows the three newest, independent of the dashboard date range. | - -**Time saved** applies a heuristic of 25 finished lines per engineer-hour to merged factory pull requests. The per-teammate view divides that estimate by your current team size. Treat it as a directional heuristic, not a financial return or a billing figure. - -**Cost per PR** shows a median headline with the mean available, and offers **By complexity** and **By size** views. **Most expensive PRs** detail can break out inference usage by model and show a high-level human-touch or autonomy indicator. Treat that indicator as descriptive context; this page does not define additional classification rules. Dollar amounts convert recorded credits using your team's billing-tier credit rate, but remain presentational and do not represent billing. +| **Total runs** | Agent runs, with breakdowns by agent type, status, source, model, and more. | Includes Warp's own measurement and improvement runs. | +| **PRs opened** | Pull requests created from factory work. | Collected differently from **PRs merged**. | +| **PRs merged** | Pull requests that merged. | Requires the GitHub App; counting starts when you install it. | +| **Autonomy %** | The share of merged work that shipped without human edits. | Based on recorded PR signals, not a quality judgment. | +| **Time saved** | A rough estimate of engineer-hours saved by merged factory PRs. | A rule of thumb for spotting trends, not a billing or ROI figure. | +| **PR latency** | How long pull requests spend in each stage. | Can be incomplete, depending on webhook coverage. | +| **Cost per PR** | An estimate of credits spent per pull request. | A lower-bound estimate. Dollar amounts are for display, not billing. | +| **Most expensive PRs** | The highest-cost pull requests. | Full detail requires the GitHub App. | +| **Scorer cards** | Results from your Scorers. | Reflects the criteria your team defined. | +| **Self-improvement PRs** | The three newest Self-improvement pull requests. | Not affected by the dashboard date range. | + +**Time saved** converts merged factory pull requests into an estimate of engineer-hours, and the per-teammate view divides that estimate by your team size. **Cost per PR** shows the median by default, plus mean, **By complexity**, and **By size** views. Treat both as directional estimates rather than financial figures. :::caution -Merged, latency, and detailed PR coverage is GitHub App-only, starts when collection begins, and is not backfilled. **PRs opened** and **PRs merged** use different data sources, so do not interpret their ratio as a merge rate. Run count includes evaluation, benchmark, and Self-improvement activity. Cost per PR and Time saved are estimates, not financial ROI calculations. +Merge, latency, and per-PR detail metrics require the GitHub App, and collection starts when you install it; earlier history isn't backfilled. **PRs opened** and **PRs merged** come from different sources, so their ratio isn't a merge rate. Run counts include evaluation, benchmark, and Self-improvement runs. ::: -Use dashboard changes to choose runs for investigation, not to infer a cause. A higher run count with a stable opened-PR count might reflect harder work, retries, measurement activity, or unclear agent instructions. +Use the dashboard to pick which runs to investigate, not to conclude what caused a change. A higher run count with a flat PR count could mean harder tasks, retries, measurement activity, or unclear agent instructions. Open the runs to find out. ## Configure Scorers -A user-defined **Scorer** tells an LLM judge how to classify completed conversations. User-defined Scorers have no built-in rubrics and currently support classification rather than numeric quality evaluation. They are separate from built-in **Correctness**, a Warp-managed scorer that benchmarks run automatically. Define one user-defined Scorer for one decision so its failures remain actionable. +A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as "did the agent run the tests before opening a PR?" Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. Benchmarks also run a separate built-in scorer, **Correctness**, which you don't configure. Configure these fields: -* **Judge instructions** - State the observable criteria the judge applies. -* **Classifications** - Define the allowed labels and their scores. -* **Pass threshold** - Determines which recorded scores pass or fail. -* **Sample rate** - Sets the portion of eligible conversations to evaluate. -* **Judge model** - Selects a supported model or an auto router. +* **Judge instructions** - The criteria the judge checks for. +* **Classifications** - The labels the judge can assign, each with a score. +* **Pass threshold** - The score a conversation needs to pass. +* **Sample rate** - The portion of eligible conversations to evaluate. +* **Judge model** - The model that acts as the judge. -You can scope a Scorer to all agents or selected agents, then keep it active or pause it. The judge evaluates eligible completed conversations after a quiet period and returns a classification, score, and reasoning. +You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after an eligible conversation completes, the judge evaluates it and records a classification, a score, and its reasoning. -| Mode | Use when | Outcome | +| Mode | Use it when | You get | | --- | --- | --- | -| Manual | Investigating one completed run or testing judge instructions. | An evaluation for the selected conversation. | -| Periodic | Monitoring a sample of eligible conversations over time. | A baseline for comparison before and after a change. | +| Manual | You want to score one conversation or test new judge instructions. | An evaluation of the selected conversation. | +| Periodic | You want an ongoing sample to track over time. | A baseline to compare against after a change. | -Changing **Pass threshold** reclassifies historical scores against the current threshold when you view them. It does not change the recorded classification, score, or reasoning. +Changing **Pass threshold** only changes how past scores display as pass or fail. The recorded classification, score, and reasoning stay the same. ## Compare configurations with benchmarks -A benchmark suite fixes one agent and compares configurations of it on the same tasks. Define the suite, then launch it with the configurations you want to test: +A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A suite includes: -* **Agent** - The single suite-level agent whose configurations you compare. -* **Tasks** - Fixed prompts with explicit success criteria that freeze for a benchmark launch. -* **Configurations** - Launch-time combinations of harness, model, and runner. -* **Scorers** - User-defined classification Scorers applied to every trial alongside built-in **Correctness**. -* **Repetitions** - Multiple trials per task and configuration. +* **Agent** - The agent whose configurations you compare. +* **Tasks** - Fixed prompts with success criteria. Task definitions freeze when you launch. +* **Configurations** - The harness, model, and runner combinations to test. +* **Scorers** - Your classification Scorers, applied to every trial. +* **Repetitions** - The number of trials per task and configuration. -You can create a benchmark task from a completed run's detail pane. Warp copies the run's source input into the task. Define its success criteria before launch so **Correctness** evaluates every trial against the frozen task definition. +You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. -Every benchmark automatically runs built-in **Correctness**, a Warp-managed scorer that returns a binary pass or fail against the task's frozen success criteria. Results include a per-configuration scoreboard, pass rate by Scorer, cost and quality scatter, and per-task comparison. **Correctness** and user-defined Scorer results provide separate quality signals; Warp Factories does not combine them into a numeric quality score or select a winning configuration. Benchmark credit totals cover recorded compute and platform usage, exclude inference usage, and do not represent full USD cost. +Every benchmark also runs **Correctness**, a built-in scorer that marks each trial as pass or fail against the task's success criteria. Results include a per-configuration scoreboard, pass rate by Scorer, a cost and quality scatter plot, and per-task comparisons. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Credit totals exclude inference usage, so they understate the full cost. -Use enough repetitions to distinguish a stable difference from one successful trial, and apply constraints the suite does not encode, including required tools, security policy, provider availability, and the severity of each classification. +Run enough repetitions to separate a real difference from one lucky trial, and weigh factors the suite doesn't capture, like required tools, security policy, provider availability, and how serious each failure type is. ## Configure and review Self-improvement -Toggle **Self-improvement** on for each Scorer whose failing results warrant automated investigation. The factory-level **Analysis model** selects the model used to analyze failures and cluster related findings. GitHub-backed, file-managed factories cannot set **Analysis model** in the control room. +Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. The factory-level **Analysis model** setting chooses the model that analyzes failures and groups related findings. Factories managed from files in a GitHub repo can't change this setting in the control room. -Self-improvement dispatches investigation workers, clusters their findings, and files ordinary follow-up factory tasks as agent runs. A follow-up run can propose a change to application code or, when the factory configuration repo is available, to prompts, skills, or configuration. Self-improvement can produce pull requests, but it does not guarantee a pull request or automatically adopt a change. +Self-improvement investigates failing results, clusters related causes, and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code, or to prompts, skills, and configuration when the factory's configuration repo is available. It can open pull requests, but a pull request isn't guaranteed, and nothing is adopted without your review. -The dashboard's **Self-improvement PRs** shows the three newest pull requests from Self-improvement, independent of the dashboard date range. Each includes a deterministic **Regressions addressed** block that links the source runs, the failed Scorer and classification, and the triage run, so you can trace a proposed change back to the evidence that prompted it. +The **Self-improvement PRs** card shows the three newest Self-improvement pull requests, regardless of the dashboard date range. Each pull request includes a **Regressions addressed** section that links the failing runs, the Scorer result, and the investigation run, so you can trace a proposed change back to its evidence. ## Run a practical improvement loop -Change one measurable part of the factory at a time: +Change one measurable thing at a time: -1. **Define a Scorer.** Choose one agent role and observable failure mode. Write **Judge instructions** and **Classifications**, then set the threshold, sample rate, judge model, scope, and state. Confirm that results match several manual reviews. -2. **Collect a baseline.** Run periodic scoring until the classification distribution represents normal work. Record Scorer settings, the date range, activity metrics, and relevant cost components. -3. **Inspect failures.** Read judge reasoning and conversations, then identify causes such as missing context, unclear instructions, unavailable tools, or incomplete validation. Enable Self-improvement only when repeated failures warrant grouped investigation. -4. **Benchmark a candidate.** Compare configurations of one agent across the same tasks and repetitions. Review the configuration scoreboard, pass rate by Scorer, cost and quality scatter, per-task results, and outliers. -5. **Review and adopt.** Decide whether the evidence supports changing a harness, model, runner, parameter, prompt, skill, or other definition. Review any Self-improvement pull request or follow-up task using the same standards as a human-authored change. -6. **Continue monitoring.** Keep the Scorer active, compare results with the baseline, and watch for regressions. Pause or revise a Scorer when its rubric no longer represents the team's requirements. +1. **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then compare the judge's results against a few conversations you review yourself. +2. **Collect a baseline.** Let periodic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs. +3. **Inspect failures.** Read the judge's reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating. +4. **Benchmark a candidate.** Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference. +5. **Review and adopt.** If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones. +6. **Keep monitoring.** Leave the Scorer active and compare new results against your baseline. Pause or revise the Scorer when its criteria no longer match what your team needs. -A narrow change with a stable benchmark and Scorer provides stronger evidence than a broad reconfiguration with several possible causes. +A small, measured change gives you stronger evidence than a broad reconfiguration with several possible causes. ## Next step From ce81d9b52dbac099825be720c75feaf2c36715ba Mon Sep 17 00:00:00 2001 From: hongyi-chen Date: Sun, 16 Aug 2026 05:33:53 +0000 Subject: [PATCH 3/4] Drop Keep in mind columns; state facts in descriptions Reviewer feedback: docs should read as objective statements, not advice. Both tables are now two columns, with measurement facts folded into the metric descriptions and coverage facts kept in the caution block. Co-Authored-By: Warp --- .../docs/factories/measure-and-improve.mdx | 40 +++++++++---------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index fd2ddb61..4548912e 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -10,31 +10,31 @@ topic: factories Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it. -| Feature | What it tells you | Keep in mind | -| --- | --- | --- | -| Dashboard metrics | How much work the factory produced, and what it cost. | Some metrics require the GitHub App, and measurement runs count as activity. | -| Scorers | Whether completed conversations meet criteria you define. | Scorers classify conversations; they don't grade quality on a numeric scale. | -| Benchmarks | How different configurations perform on the same tasks. | Results don't pick a winner or include full cost. | -| Self-improvement | Which repeated failures need follow-up work. | It can open pull requests, but you review and adopt every change yourself. | +| Feature | What it tells you | +| --- | --- | +| Dashboard metrics | How much work the factory produced, and what it cost. | +| Scorers | Whether completed conversations meet criteria you define. | +| Benchmarks | How different configurations perform on the same tasks. | +| Self-improvement | Which repeated failures get investigated and turned into follow-up work. | ## Read dashboard metrics The control room dashboard shows activity, cost, autonomy, and evaluation results: -| Metric | What it shows | Keep in mind | -| --- | --- | --- | -| **Total runs** | Agent runs, with breakdowns by agent type, status, source, model, and more. | Includes Warp's own measurement and improvement runs. | -| **PRs opened** | Pull requests created from factory work. | Collected differently from **PRs merged**. | -| **PRs merged** | Pull requests that merged. | Requires the GitHub App; counting starts when you install it. | -| **Autonomy %** | The share of merged work that shipped without human edits. | Based on recorded PR signals, not a quality judgment. | -| **Time saved** | A rough estimate of engineer-hours saved by merged factory PRs. | A rule of thumb for spotting trends, not a billing or ROI figure. | -| **PR latency** | How long pull requests spend in each stage. | Can be incomplete, depending on webhook coverage. | -| **Cost per PR** | An estimate of credits spent per pull request. | A lower-bound estimate. Dollar amounts are for display, not billing. | -| **Most expensive PRs** | The highest-cost pull requests. | Full detail requires the GitHub App. | -| **Scorer cards** | Results from your Scorers. | Reflects the criteria your team defined. | -| **Self-improvement PRs** | The three newest Self-improvement pull requests. | Not affected by the dashboard date range. | - -**Time saved** converts merged factory pull requests into an estimate of engineer-hours, and the per-teammate view divides that estimate by your team size. **Cost per PR** shows the median by default, plus mean, **By complexity**, and **By size** views. Treat both as directional estimates rather than financial figures. +| Metric | What it shows | +| --- | --- | +| **Total runs** | All agent runs, with breakdowns by agent type, status, source, model, and more. | +| **PRs opened** | Pull requests created from factory work. | +| **PRs merged** | Pull requests that merged, collected through the GitHub App. | +| **Autonomy %** | The share of merged work that shipped without human edits, based on recorded PR signals. | +| **Time saved** | An estimate of engineer-hours saved by merged factory PRs. | +| **PR latency** | How long pull requests spend in each stage, based on webhook data. | +| **Cost per PR** | A lower-bound estimate of credits spent per pull request. | +| **Most expensive PRs** | The highest-cost pull requests. Full detail requires the GitHub App. | +| **Scorer cards** | Results from your Scorers. | +| **Self-improvement PRs** | The three newest Self-improvement pull requests, regardless of the selected date range. | + +**Time saved** and **Cost per PR** are estimates, not billing figures. Time saved converts merged pull requests into engineer-hours, and the per-teammate view divides that estimate by your team size. Cost per PR shows the median by default, plus mean, **By complexity**, and **By size** views; dollar amounts convert credits at your team's billing rate for display only. :::caution Merge, latency, and per-PR detail metrics require the GitHub App, and collection starts when you install it; earlier history isn't backfilled. **PRs opened** and **PRs merged** come from different sources, so their ratio isn't a merge rate. Run counts include evaluation, benchmark, and Self-improvement runs. From f7e89253be8de049d1f878e001cddcf873f74963 Mon Sep 17 00:00:00 2001 From: hongyi-chen Date: Sun, 16 Aug 2026 05:41:52 +0000 Subject: [PATCH 4/4] Remove remaining implementation details and duplication - Metric rows describe only what each metric shows; GitHub App coverage facts live in the caution block alone - Cut UI inventory (median/mean/By complexity/By size views, scoreboard, scatter plot) and billing-rate conversion detail - Replace jargon: eligible, lower-bound, freeze, clusters, PR signals, webhook data - Dedupe Self-improvement paragraphs and the three-newest-PRs fact, fix the ambiguous pronoun, drop the closing aphorism and generic benchmark advice already covered by loop step 4 Co-Authored-By: Warp --- .../docs/factories/measure-and-improve.mdx | 38 +++++++++---------- 1 file changed, 17 insertions(+), 21 deletions(-) diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index 4548912e..4d95c2a5 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -25,67 +25,65 @@ The control room dashboard shows activity, cost, autonomy, and evaluation result | --- | --- | | **Total runs** | All agent runs, with breakdowns by agent type, status, source, model, and more. | | **PRs opened** | Pull requests created from factory work. | -| **PRs merged** | Pull requests that merged, collected through the GitHub App. | -| **Autonomy %** | The share of merged work that shipped without human edits, based on recorded PR signals. | +| **PRs merged** | Pull requests that merged. | +| **Autonomy %** | The share of merged work that shipped without human edits. | | **Time saved** | An estimate of engineer-hours saved by merged factory PRs. | -| **PR latency** | How long pull requests spend in each stage, based on webhook data. | -| **Cost per PR** | A lower-bound estimate of credits spent per pull request. | -| **Most expensive PRs** | The highest-cost pull requests. Full detail requires the GitHub App. | +| **PR latency** | How long pull requests spend in each stage. | +| **Cost per PR** | An estimate of credits spent per pull request. | +| **Most expensive PRs** | The highest-cost pull requests. | | **Scorer cards** | Results from your Scorers. | | **Self-improvement PRs** | The three newest Self-improvement pull requests, regardless of the selected date range. | -**Time saved** and **Cost per PR** are estimates, not billing figures. Time saved converts merged pull requests into engineer-hours, and the per-teammate view divides that estimate by your team size. Cost per PR shows the median by default, plus mean, **By complexity**, and **By size** views; dollar amounts convert credits at your team's billing rate for display only. +**Time saved** and **Cost per PR** are estimates, not billing figures. Time saved converts merged pull requests into engineer-hours, and the per-teammate view divides that estimate by your team size. Cost per PR counts recorded credits and can undercount actual usage. :::caution -Merge, latency, and per-PR detail metrics require the GitHub App, and collection starts when you install it; earlier history isn't backfilled. **PRs opened** and **PRs merged** come from different sources, so their ratio isn't a merge rate. Run counts include evaluation, benchmark, and Self-improvement runs. +**PRs merged**, **PR latency**, and the detail in **Most expensive PRs** require the GitHub App and only cover activity from after you install it. The ratio of **PRs merged** to **PRs opened** isn't a merge rate; the two are counted differently. Run counts include evaluation, benchmark, and Self-improvement runs. ::: -Use the dashboard to pick which runs to investigate, not to conclude what caused a change. A higher run count with a flat PR count could mean harder tasks, retries, measurement activity, or unclear agent instructions. Open the runs to find out. +Use the dashboard to pick which runs to investigate, not to conclude what caused a change. A higher run count with a flat PR count could mean harder tasks, retries, or measurement activity. ## Configure Scorers -A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as "did the agent run the tests before opening a PR?" Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. Benchmarks also run a separate built-in scorer, **Correctness**, which you don't configure. +A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as "did the agent run the tests before opening a PR?" Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. Configure these fields: * **Judge instructions** - The criteria the judge checks for. * **Classifications** - The labels the judge can assign, each with a score. * **Pass threshold** - The score a conversation needs to pass. -* **Sample rate** - The portion of eligible conversations to evaluate. +* **Sample rate** - The portion of completed conversations to evaluate. * **Judge model** - The model that acts as the judge. -You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after an eligible conversation completes, the judge evaluates it and records a classification, a score, and its reasoning. +You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after a conversation completes, the judge evaluates it and records a classification, a score, and its reasoning. | Mode | Use it when | You get | | --- | --- | --- | | Manual | You want to score one conversation or test new judge instructions. | An evaluation of the selected conversation. | | Periodic | You want an ongoing sample to track over time. | A baseline to compare against after a change. | -Changing **Pass threshold** only changes how past scores display as pass or fail. The recorded classification, score, and reasoning stay the same. +Changing **Pass threshold** updates how past scores display as pass or fail; the recorded results don't change. ## Compare configurations with benchmarks A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A suite includes: * **Agent** - The agent whose configurations you compare. -* **Tasks** - Fixed prompts with success criteria. Task definitions freeze when you launch. +* **Tasks** - Fixed prompts with success criteria. * **Configurations** - The harness, model, and runner combinations to test. * **Scorers** - Your classification Scorers, applied to every trial. * **Repetitions** - The number of trials per task and configuration. You can create a benchmark task from a completed run's detail pane, and Warp copies the run's input into the task. Add success criteria before you launch. -Every benchmark also runs **Correctness**, a built-in scorer that marks each trial as pass or fail against the task's success criteria. Results include a per-configuration scoreboard, pass rate by Scorer, a cost and quality scatter plot, and per-task comparisons. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Credit totals exclude inference usage, so they understate the full cost. - -Run enough repetitions to separate a real difference from one lucky trial, and weigh factors the suite doesn't capture, like required tools, security policy, provider availability, and how serious each failure type is. +Every benchmark also runs **Correctness**, a built-in scorer that marks each trial as pass or fail against the task's success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn't combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don't include model usage, so the true cost is higher. ## Configure and review Self-improvement -Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. The factory-level **Analysis model** setting chooses the model that analyzes failures and groups related findings. Factories managed from files in a GitHub repo can't change this setting in the control room. +Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code, or to the factory's prompts, skills, and configuration when that repo is available. Follow-up runs can open pull requests, but nothing is adopted without your review. -Self-improvement investigates failing results, clusters related causes, and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code, or to prompts, skills, and configuration when the factory's configuration repo is available. It can open pull requests, but a pull request isn't guaranteed, and nothing is adopted without your review. +The factory-level **Analysis model** setting chooses the model used for this analysis. Factories managed from files in a GitHub repo can't change this setting in the control room. -The **Self-improvement PRs** card shows the three newest Self-improvement pull requests, regardless of the dashboard date range. Each pull request includes a **Regressions addressed** section that links the failing runs, the Scorer result, and the investigation run, so you can trace a proposed change back to its evidence. +Each Self-improvement pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence. ## Run a practical improvement loop @@ -98,8 +96,6 @@ Change one measurable thing at a time: 5. **Review and adopt.** If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones. 6. **Keep monitoring.** Leave the Scorer active and compare new results against your baseline. Pause or revise the Scorer when its criteria no longer match what your team needs. -A small, measured change gives you stronger evidence than a broad reconfiguration with several possible causes. - ## Next step Record an adopted change in [factory definitions as code](./factory-as-code) so your team can review the factory configuration.