feat: add cancelled run state and fix extension pause/resume flow

[achuvyas-kv]

Problem

Stopping a run in the browser extension left the user with nothing useful:

• Cancelling discarded everything. Clicking Stop between evaluators wiped the queue and dropped back to the idle screen — no report, no record of the evaluators that had already completed.
• Cancelled evaluators were reported as FAIL. When a partial result did survive, it was hardcoded to FAIL, so a run the user interrupted was indistinguishable from a genuine vulnerability.
• Cancelling still ran the LLM judge. The interrupted evaluator was judged on its partial transcript, which added seconds to a stop the user wanted immediate and produced a full PASS/FAIL-style rationale, score, and confidence for a turn the target never finished.
• Cancelled evaluators diluted the safety score. They contributed their severity weight to the denominator but never to the numerator, so 1 pass + 1 cancelled scored well under 100%.
• Pause and Stop were indistinguishable. Both arrive as the same OPFOR_STOP signal, and the stop handler did both things at once — wrote a terminal CANCELLED result and saved a resumable snapshot. Everything downstream then guessed, and guessed inconsistently:
  • A merely-paused evaluator surfaced as CANCELLED in the report.
  • A genuine Stop still left a resumable snapshot, so reopening the extension offered a stale "Run paused / Resume" screen.
  • Resuming showed no progress (0 turns) and restarted the turn counter at 1.
  • Resuming after closing the popup dropped every evaluator after the paused one.
  • A previous evaluator's completed result could resolve the next evaluator instantly, with the wrong verdict.
• No feedback on Pause/Stop. Both take a few seconds while the in-flight evaluator unwinds, with no indication anything happened — Pause in particular looked broken.

Solution

Make cancellation a real, first-class outcome, and give Pause and Stop opposite, explicit contracts.

Cancellation is its own state. A new CANCELLED verdict sits alongside PASS/FAIL, styled with the existing amber --warn token — same card, pill, and banner layout, only a third colour. Cancelling now always produces a report of the evaluators completed up to that point.

Never judge what was cancelled. The judge call is skipped entirely on cancel. Because an unfinished evaluator has no verdict, no score, confidence, or reasoning is fabricated for it — the report shows — rather than a plausible-looking number.

Pause and Stop carry explicit intent. The popup tags the OPFOR_UI_STOP message with intent: "pause" | "cancel", and a single finalizeUserInterruption() helper in the background enforces the split, so the two can't drift apart:

	resumable snapshot	terminal result
Pause	✅ saved	❌ none — an unfinished evaluator has no verdict
Stop	❌ removed	✅ CANCELLED

The reply carries the intent back, so the popup never infers pause-vs-cancel from the ambiguous paused flag.

Changes

All in runners/extension/ — no core or CLI changes.

File	What changed
orchestrator.js	Added finalizeUserInterruption() + cancelledJudgment(); both stop paths route through them. Removed the judge call on stop (and the now-unused judgeResponse import). Carry the restored transcript and resumedFromRound through setRunStatus on resume instead of writing []. Clear opforLastResult at the start of every run, including resume. Degrade an unresumable pause to a cancel.
popup.js	CANCELLED verdict throughout renderDone, buildReport, generateHtmlReport, and run history. Exclude cancelled evaluators from the weighted score and add a judged count. null score/confidence for cancelled rows. Added userInterrupted() and guarded all three OPFOR_JUDGE_PARTIAL call sites. Added a fast interrupt path to the storage poller. Ownership-check persisted records via matchesEvaluator(). Park the queue on pause instead of clearing it; restore the full queue in checkPausedRun. Added renderPausedScreen() and setPauseBusy/setStopBusy.
popup.html	CANCELLED styling for the verdict card, result rows/pills, and history stripe. Spinner + label markup for Pause/Stop, :disabled styles, and fixed-width labels so the icon doesn't shift when the label swaps.
service_worker.js	Record state.OPFOR_STOP_INTENT from the stop message.
state.js	New OPFOR_STOP_INTENT field, defaulting to "cancel" so an untagged stop never leaves a resumable run behind.
domTarget.js	New roundOffset option so round numbers continue after a resume rather than restarting at 1.

Notable bug found along the way

The verdict check keyed off result.stopped, but stopped: true is also set on legitimately-judged "recovered" partials — so runs recovered after an interruption were being mislabelled CANCELLED and losing their real verdict. All three sites now key off stopReason === "user_stop".

Issue

N/A

How to test

popup.js / popup.html are loaded directly by the manifest, so no build step is needed — but the background service worker does need a full reload: chrome://extensions → reload icon. Reopening the popup alone is not enough.

Cancel produces a report

1. Start a suite with 2–3 evaluators. Let the first finish, then click Stop mid-way through the second.
2. Stop shows a spinner and "Stopping…" immediately, with the icon staying put (no left-shift), and resolves in about a second — no judge call.
3. The done screen shows aggregate CANCELLED in amber, the completed evaluator with its real verdict, and the interrupted one as CANCELLED.
4. The interrupted row shows — for score and confidence, and reads "Cancelled by user before this evaluator could finish." — not an LLM rationale.
5. Safety Score reads "Based on 1 evaluator · 1 cancelled" and is not diluted by the cancelled one.
6. Download the report — the same states appear in the exported HTML.

Stop does not leave a resumable run
7. Close and reopen the extension. It should return to idle — not a "Run paused / Resume" screen.

Pause and resume
8. Start a 3-evaluator suite. Let the first finish, click Pause during the second.
9. Pause shows its own spinner and "Pausing…", then the paused screen reports real progress, e.g. evaluator 2 of 3 · 3 of 10 turns done · saved.
10. Close and reopen the popup, then click Resume.
11. The running screen shows the prior turn count (not 0) and the counter continues (e.g. turn 4), rather than restarting at 1.
12. The run continues through evaluator 3 — the rest of the suite is not dropped.
13. The final report shows the paused evaluator with its real PASS/FAIL verdict, never CANCELLED, and no evaluator resolves instantly with a previous evaluator's verdict.

Edge cases
14. Pause, then click Stop from the paused screen → a CANCELLED report of what completed, and no resumable run on reopen.
15. Pause, then Discard → idle, with no resumable run offered on reopen.

Summary by CodeRabbit

• New Features

  • Added distinct Pause vs Cancel handling for evaluator runs.
  • Cancelled runs now surface a clear CANCELLED state across the UI, reports, and history.
  • Cancelled evaluators are excluded from safety/attack scoring, with cancelled verdicts reflected in generated HTML.
  • Paused runs can resume with restored progress and queued evaluations.

• Bug Fixes

  • Improved round numbering alignment when resuming runs.
  • Strengthened interruption finalization to prevent inconsistent run/report outcomes.
  • Reduced background polling race issues and ensured correct cancellation vs pause behavior.

[coderabbitai]

Walkthrough

The extension now distinguishes pause and cancellation intents, persists resumable or cancelled outcomes, restores paused evaluator queues and resumed round numbering, and presents CANCELLED results across popup screens, reports, and history.

Changes

Interruption lifecycle

Layer / File(s)	Summary
Interruption intent and orchestration runners/extension/state.js, runners/extension/service_worker.js, runners/extension/orchestrator.js, runners/extension/domTarget.js	Pause or cancel intent is propagated through shared state and service-worker messages; orchestration finalizes interruptions, restores bounded transcript state, and applies resumed round offsets.
Popup interruption and recovery runners/extension/popup.js	Pause parks the evaluator queue, cancellation discards it, controls show busy states, polling accounts for interruption races and ownership, and resumed runs restore persisted progress.
Cancelled verdict and report handling runners/extension/popup.js	Cancelled evaluators are excluded from weighted scoring and confidence calculations, while reports, HTML output, summaries, and history classify cancellation separately.
Cancelled control and status presentation runners/extension/popup.html	Pause/Stop controls gain stable labels, spinners, and disabled states; cancelled verdicts receive dedicated styling across screens, evaluator results, pills, and history.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Popup
  participant ServiceWorker
  participant Orchestrator
  participant Storage
  Popup->>ServiceWorker: Send OPFOR_UI_STOP with pause or cancel intent
  ServiceWorker->>Orchestrator: Set stop state and intent
  Orchestrator->>Storage: Persist paused snapshot or cancelled result
  Storage-->>Popup: Return interruption status
  Popup->>Popup: Render paused or CANCELLED state

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main changes: cancelled runs and pause/resume fixes.
Description check	✅ Passed	The description covers Problem, Solution, Changes, Issue, and How to test; only Screenshots is omitted, which is acceptable if not applicable.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/extension-cancelled-run-reports

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@runners/extension/orchestrator.js`:
- Around line 672-683: The resumed-round field is written as resumedFromRound by
setRunStatus but read as lastRound by the popup. In
runners/extension/orchestrator.js:672-683 and
runners/extension/popup.js:3476-3483, align the writer and reader on one field
name—prefer updating the popup recovery logic to read
opforRunStatus.resumedFromRound if the orchestrator schema remains unchanged—so
resumed runs use priorRounds instead of deriving the counter from the truncated
transcript.
- Around line 898-921: Update the degrade-to-cancel branch in
finalizeUserInterruption to remove any existing opforPausedRun after persisting
the partial result and before marking the response cancelled. Reuse the same
pause-state cleanup mechanism used by the normal cancel path, ensuring stale
resume data cannot remain.

In `@runners/extension/popup.html`:
- Line 2416: Update the pause and stop spinner wrappers in the button markup
from div elements to span elements, and mark both spinner elements as decorative
while preserving their existing classes and hidden state.

In `@runners/extension/popup.js`:
- Around line 1700-1706: Use the existing judged-count fallback used by
renderHistoryList when generating the report summary, and replace direct
summary.judged references in the summary text and breach-rate card (including
the lines around 1700, 1706, and 1722) with judgedCount. Preserve the existing
wording and calculations while ensuring reports persisted before this PR display
a numeric evaluator count.
- Around line 1108-1126: Update the summary calculation around safetyScore and
the rendering that consumes it so no judged evaluators produces a null/unknown
score instead of 0. Preserve numeric scoring when weightedTotal is nonzero, and
render an em dash rather than a red 0% bar when summary.judged is 0.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 766f6fba-226b-45d1-85a7-7753d2eaad3c

📥 Commits

Reviewing files that changed from the base of the PR and between e45a62c and 3668640.

📒 Files selected for processing (6)

• runners/extension/domTarget.js
• runners/extension/orchestrator.js
• runners/extension/popup.html
• runners/extension/popup.js
• runners/extension/service_worker.js
• runners/extension/state.js

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@runners/extension/popup.js`:
- Around line 1284-1290: Update the empty-findings message in the report to use
a neutral unassessed/cancelled message when judgedCount(summary) is zero, and
retain the existing “passed all evaluated attack patterns” message only when at
least one evaluator was judged. Reuse the existing judged value and keep the
riskLevel behavior unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0e9999d7-e93b-4994-8469-25f168b5dfef

📥 Commits

Reviewing files that changed from the base of the PR and between 3668640 and 0537ed7.

📒 Files selected for processing (3)

• runners/extension/orchestrator.js
• runners/extension/popup.html
• runners/extension/popup.js

🚧 Files skipped from review as they are similar to previous changes (2)

• runners/extension/popup.html
• runners/extension/orchestrator.js