Skip to content

FIX: CLI Usability Updates#2194

Merged
rlundeen2 merged 1 commit into
microsoft:mainfrom
rlundeen2:rlundeen2-supreme-system
Jul 15, 2026
Merged

FIX: CLI Usability Updates#2194
rlundeen2 merged 1 commit into
microsoft:mainfrom
rlundeen2:rlundeen2-supreme-system

Conversation

@rlundeen2

Copy link
Copy Markdown
Contributor

Summary

A set of CLI usability and diagnostic improvements for scenario runs (pyrit_scan), making progress reporting honest and errors actionable.

Changes

  • Accurate technique progress count. ScenarioResult.get_techniques_used() now aggregates atomic-attack cells by display group, so the progress bar and summary show real technique counts (e.g. 2/14) instead of atomic-attack-cell counts that could exceed the total.
  • Fix adversarial-chat resolution for simulated-conversation techniques. New AttackTechniqueFactory.resolve_adversarial_chat() lazily resolves the default adversarial target when a technique's seed group carries a simulated conversation, fixing adversarial_chat is required when seed_group has a simulated conversation config failures for techniques like many_shot_*.
  • More actionable errors. Atomic-attack execution errors now include the atomic-attack name.
  • Surface failures and retry pressure in the summary even on non-failing runs: per-failed-attack error type/message and total retry counts. Dropped the redundant Completed line (it always equalled Total Attacks).
  • Stream retry warnings during polling. Each attack's retries are printed once as its result lands, including the component role/name and endpoint captured on the RetryEvents.
  • --start-server output. Prints the log path on success and tails the log on crash/timeout.

Testing

  • pre-commit clean (ruff format/check, ty).
  • Affected unit suites pass (342 tests): CLI output/scan/server-launcher, scenario-run service, scenario result, technique factory, matrix builder.

Several UX/diagnostic improvements to `pyrit_scan` output and scenario execution:

- Fix technique progress count: `get_techniques_used()` now aggregates atomic-attack
  cells by display group, so the progress bar and summary report real technique counts
  (e.g. 2/14) instead of atomic-attack-cell counts that could exceed the total.
- Resolve adversarial chat lazily for simulated-conversation techniques: add
  `AttackTechniqueFactory.resolve_adversarial_chat()` so matrix-built atomic attacks
  get the default adversarial target when a seed group carries a simulated
  conversation, fixing "adversarial_chat is required..." failures.
- Include the atomic-attack name in execution error messages for easier triage.
- Surface failed attacks and retry pressure in the run summary even on non-failing
  runs (error type/message per failed attack, total retries), and drop the redundant
  "Completed" line.
- Stream per-attack retry warnings as each result lands during polling, including
  component role/name and endpoint from the captured RetryEvents.
- `--start-server` now prints the log path on success and tails the log on
  crash/timeout.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 35659834-769b-4088-93cc-bd2603dcc65c
Comment thread pyrit/cli/_output.py
@rlundeen2 rlundeen2 added this pull request to the merge queue Jul 15, 2026
Merged via the queue into microsoft:main with commit 6d3442b Jul 15, 2026
52 checks passed
@rlundeen2 rlundeen2 deleted the rlundeen2-supreme-system branch July 15, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants