Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 0 additions & 16 deletions .pyrit_conf_example
Original file line number Diff line number Diff line change
Expand Up @@ -53,22 +53,6 @@ initializers:
- name: technique
- name: load_default_datasets

# Default Scenario
# ----------------
# Optional default scenario to run when invoking `pyrit_scan` without a
# positional scenario name. Also accepts custom parameters declared by the
# scenario via its `supported_parameters()` classmethod (see
# `pyrit_scan <scenario> --help` to discover them).
#
# - name: scenario registry name (e.g. `airt.scam`, `foundry.red_team_agent`)
# - args: per-scenario custom parameters; CLI flags override these per key
#
# Example:
# scenario:
# name: airt.scam
# args:
# max_turns: 10

# Operator and Operation Labels
# ------------------------------
# Default labels applied to all attacks created with PyRIT.
Expand Down
1 change: 0 additions & 1 deletion doc/code/datasets/5_simulated_conversation.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,6 @@
"simulated_conversation = SeedGroup(seeds=simulated_conversation_prompts)\n",
"\n",
"# View the conversation prefix (N-1 turns)\n",
"# Note: print_messages_async was deprecated. Using output_attack_async instead.\n",
"# For direct message printing, use: from pyrit.output import output_conversation_async\n",
"\n",
"print(f\"\\nPrepended conversation messages: {len(simulated_conversation.prepended_conversation)}\")\n",
Expand Down
1 change: 0 additions & 1 deletion doc/code/datasets/5_simulated_conversation.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,6 @@
simulated_conversation = SeedGroup(seeds=simulated_conversation_prompts)

# View the conversation prefix (N-1 turns)
# Note: print_messages_async was deprecated. Using output_attack_async instead.
# For direct message printing, use: from pyrit.output import output_conversation_async

print(f"\nPrepended conversation messages: {len(simulated_conversation.prepended_conversation)}")
Expand Down
3 changes: 0 additions & 3 deletions doc/code/memory/1_sqlite_memory.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,6 @@
" scorer_class_identifier: JSON NOT NULL\n",
" prompt_request_response_id: CHAR(32) NULL\n",
" timestamp: DATETIME NOT NULL\n",
" task: VARCHAR NULL\n",
" objective: VARCHAR NULL\n",
" pyrit_version: VARCHAR NULL\n",
"\n",
Expand Down Expand Up @@ -111,7 +110,6 @@
" outcome_reason: VARCHAR NULL\n",
" attack_metadata: JSON NULL\n",
" labels: JSON NULL\n",
" targeted_harm_categories: JSON NULL\n",
" pruned_conversation_ids: JSON NULL\n",
" adversarial_chat_conversation_ids: JSON NULL\n",
" timestamp: DATETIME NOT NULL\n",
Expand All @@ -135,7 +133,6 @@
" objective_target_identifier: JSON NOT NULL\n",
" objective_scorer_identifier: JSON NULL\n",
" scenario_run_state: VARCHAR NOT NULL DEFAULT ScalarElementColumnDefault('CREATED')\n",
" attack_results_json: VARCHAR NOT NULL\n",
" display_group_map_json: VARCHAR NULL\n",
" labels: JSON NULL\n",
" number_tries: INTEGER NOT NULL DEFAULT ScalarElementColumnDefault(0)\n",
Expand Down
5 changes: 2 additions & 3 deletions doc/code/memory/3_memory_data_types.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,20 +108,19 @@ Seeds are organized into [`SeedGroup`](../../../pyrit/models/seeds/seed_group.py
- **`score_rationale`**: Explanation of why the score was assigned
- **`scorer_class_identifier`**: Information about the scorer that generated this score
- **`message_piece_id`**: The ID of the piece/response being scored
- **`task`**: The original attacker's objective being evaluated
- **`objective`**: The original attacker's objective being evaluated
- **`score_metadata`**: Custom metadata specific to the scorer

Scores enable automated evaluation of attack success, content harmfulness, and other metrics throughout PyRIT's red teaming workflows.

## AttackResults

[`AttackResult`](../../../pyrit/models/attack_result.py) objects encapsulate the complete outcome of an attack execution, including metrics, evidence, and success determination. When an attack is run, the AttackResult is added to the database and can be queried later.
[`AttackResult`](../../../pyrit/models/results/attack_result.py) objects encapsulate the complete outcome of an attack execution, including metrics, evidence, and success determination. When an attack is run, the AttackResult is added to the database and can be queried later.

**Key Fields:**

- **`conversation_id`**: The conversation that produced this result
- **`objective`**: Natural-language description of the attacker's goal
- **`attack_identifier`**: `ComponentIdentifier` identifying the attack strategy used
- **`atomic_attack_identifier`**: Composite `ComponentIdentifier` combining the attack technique with seed identifiers from the dataset (see [ComponentIdentifiers](#componentidentifiers) below)
- **`last_response`**: The final `MessagePiece` generated in the attack
- **`last_score`**: The final score assigned to the last response
Expand Down
3 changes: 0 additions & 3 deletions doc/code/memory/6_azure_sql_memory.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@
" Column id (UNIQUEIDENTIFIER)\n",
" Column conversation_id (VARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column objective (NVARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column attack_identifier (NVARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column objective_sha256 (VARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column last_response_id (UNIQUEIDENTIFIER)\n",
" Column last_score_id (UNIQUEIDENTIFIER)\n",
Expand All @@ -69,7 +68,6 @@
" Column prompt_metadata (VARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column converter_identifiers (NVARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column prompt_target_identifier (NVARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column attack_identifier (NVARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column response_error (VARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column original_value_data_type (VARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column original_value (NVARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
Expand All @@ -89,7 +87,6 @@
" Column scorer_class_identifier (NVARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column prompt_request_response_id (UNIQUEIDENTIFIER)\n",
" Column timestamp (DATETIME)\n",
" Column task (VARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
" Column objective (VARCHAR COLLATE \"SQL_Latin1_General_CP1_CI_AS\")\n",
"Schema for EmbeddingData:\n",
" Column id (UNIQUEIDENTIFIER)\n",
Expand Down
2 changes: 1 addition & 1 deletion doc/code/memory/9_schema_diagram.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ flowchart LR
Sc_score_metadata["score_metadata (VARCHAR)"]
Sc_scorer_class_identifier["scorer_class_identifier (VARCHAR)"]
Sc_timestamp["timestamp (TIMESTAMP)"]
Sc_task["task (VARCHAR)"]
Sc_objective["objective (VARCHAR)"]
end
S_value_sha256 -- N:N relationship to query --> P_original_value_sha256
P_id -- 1:N relationship to query --> Sc_prompt_request_response_id
Expand Down
52 changes: 7 additions & 45 deletions doc/code/scenarios/2_custom_scenario_parameters.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
"\n",
"Sometimes a scenario needs a custom parameter that a user can set without\n",
"editing source code (`max_turns`, dataset names, feature flags, etc.).\n",
"Scenarios can declare typed parameters that flow from CLI flags or YAML\n",
"config into `self.params`.\n",
"Scenarios can declare typed parameters that flow from CLI flags into\n",
"`self.params`.\n",
"\n",
"This is different from [Common Scenario Parameters](./1_common_scenario_parameters.ipynb),\n",
"which covers the framework-level configuration surface (datasets, techniques,\n",
Expand All @@ -28,7 +28,7 @@
"```python\n",
"@classmethod\n",
"def additional_parameters(cls) -> list[Parameter]:\n",
" \"\"\"Declare custom parameters this scenario accepts from the CLI / config file.\"\"\"\n",
" \"\"\"Declare custom parameters this scenario accepts from the CLI.\"\"\"\n",
" return [\n",
" Parameter(\n",
" name=\"max_turns\",\n",
Expand Down Expand Up @@ -96,7 +96,7 @@
"source": [
"Each declaration lives inside the scenario class body, in the\n",
"`additional_parameters()` classmethod. End users don't construct `Parameter`\n",
"objects themselves; they pass values via CLI flags or YAML config.\n",
"objects themselves; they pass values via CLI flags.\n",
"\n",
"Each `Parameter` carries:\n",
"\n",
Expand Down Expand Up @@ -216,46 +216,8 @@
"pyrit_scan airt.scam --help\n",
"# ...\n",
"# --max-turns MAX_TURNS Conversation turn cap\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "6",
"metadata": {},
"source": [
"## Setting a parameter from a YAML config file\n",
"\n",
"A `scenario:` block names the scenario and supplies parameter values. CLI\n",
"flags override matching keys; absent keys fall back to YAML, then to the\n",
"declared default. See [.pyrit_conf_example](../../../.pyrit_conf_example)\n",
"for a complete config file with this and other supported sections.\n",
"\n",
"```yaml\n",
"# ~/.pyrit/.pyrit_conf\n",
"scenario:\n",
" name: airt.scam\n",
" args:\n",
" max_turns: 10\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "7",
"metadata": {},
"source": [
"A few invocation shapes from the CLI:\n",
"\n",
"```bash\n",
"pyrit_scan --config-file my_config.yaml # config provides scenario name\n",
"pyrit_scan airt.scam --config-file my_config.yaml # CLI confirms the name\n",
"pyrit_scan airt.scam --config-file my_config.yaml --max-turns 7 # CLI args win per-key\n",
"```\n",
"\n",
"`pyrit_shell` supports the YAML form when the scenario name is supplied\n",
"explicitly (`run airt.scam ...`).\n",
"\n",
"## Discovering parameters via --list-scenarios\n",
"\n",
"`--list-scenarios` prints declared parameters alongside each scenario's\n",
Expand All @@ -266,7 +228,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "8",
"id": "6",
"metadata": {},
"outputs": [
{
Expand Down Expand Up @@ -469,7 +431,7 @@
},
{
"cell_type": "markdown",
"id": "9",
"id": "7",
"metadata": {},
"source": [
"Notice the `Supported Parameters:` section under `airt.scam`. It's absent\n",
Expand Down Expand Up @@ -508,7 +470,7 @@
},
{
"cell_type": "markdown",
"id": "10",
"id": "8",
"metadata": {},
"source": [
"`Scam.max_turns` was previously hardcoded to `5` in\n",
Expand Down
36 changes: 4 additions & 32 deletions doc/code/scenarios/2_custom_scenario_parameters.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@
#
# Sometimes a scenario needs a custom parameter that a user can set without
# editing source code (`max_turns`, dataset names, feature flags, etc.).
# Scenarios can declare typed parameters that flow from CLI flags or YAML
# config into `self.params`.
# Scenarios can declare typed parameters that flow from CLI flags into
# `self.params`.
#
# This is different from [Common Scenario Parameters](./1_common_scenario_parameters.ipynb),
# which covers the framework-level configuration surface (datasets, techniques,
Expand All @@ -32,7 +32,7 @@
# ```python
# @classmethod
# def additional_parameters(cls) -> list[Parameter]:
# """Declare custom parameters this scenario accepts from the CLI / config file."""
# """Declare custom parameters this scenario accepts from the CLI."""
# return [
# Parameter(
# name="max_turns",
Expand Down Expand Up @@ -62,7 +62,7 @@
# %% [markdown]
# Each declaration lives inside the scenario class body, in the
# `additional_parameters()` classmethod. End users don't construct `Parameter`
# objects themselves; they pass values via CLI flags or YAML config.
# objects themselves; they pass values via CLI flags.
#
# Each `Parameter` carries:
#
Expand Down Expand Up @@ -155,34 +155,6 @@
# # ...
# # --max-turns MAX_TURNS Conversation turn cap
# ```

# %% [markdown]
# ## Setting a parameter from a YAML config file
#
# A `scenario:` block names the scenario and supplies parameter values. CLI
# flags override matching keys; absent keys fall back to YAML, then to the
# declared default. See [.pyrit_conf_example](../../../.pyrit_conf_example)
# for a complete config file with this and other supported sections.
#
# ```yaml
# # ~/.pyrit/.pyrit_conf
# scenario:
# name: airt.scam
# args:
# max_turns: 10
# ```

# %% [markdown]
# A few invocation shapes from the CLI:
#
# ```bash
# pyrit_scan --config-file my_config.yaml # config provides scenario name
# pyrit_scan airt.scam --config-file my_config.yaml # CLI confirms the name
# pyrit_scan airt.scam --config-file my_config.yaml --max-turns 7 # CLI args win per-key
# ```
#
# `pyrit_shell` supports the YAML form when the scenario name is supplied
# explicitly (`run airt.scam ...`).
#
# ## Discovering parameters via --list-scenarios
#
Expand Down
2 changes: 1 addition & 1 deletion doc/scanner/airt.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@
"\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
"\u001b[1m 📋 Scenario Details\u001b[0m\n",
"\u001b[36m • Name: Psychosocial\u001b[0m\n",
"\u001b[36m • Scenario Version: 1\u001b[0m\n",
"\u001b[36m • Scenario Version: 2\u001b[0m\n",
"\u001b[36m • PyRIT Version: 0.15.0.dev0\u001b[0m\n",
"\u001b[36m • Description:\u001b[0m\n",
"\u001b[36m Psychosocial Harms Scenario implementation for PyRIT. This scenario contains various psychosocial harm-based\u001b[0m\n",
Expand Down
4 changes: 2 additions & 2 deletions pyrit/backend/models/attacks.py
Original file line number Diff line number Diff line change
Expand Up @@ -479,8 +479,8 @@ class AddMessageRequest(BaseModel):
)
labels: dict[str, str] | None = Field(
None,
description="Labels to attach to every message piece. "
"Falls back to labels from existing pieces in the conversation.",
description="Request labels used for attack-level consistency checks. "
"When present, the operator must match the attack result's operator.",
)


Expand Down
Loading