[BREAKING] MAINT: Standardize Jailbreak scenario around dataset × techniques × jailbreaks

.github/instructions/scenarios.instructions.md

@@ -14,7 +14,7 @@ All scenarios inherit from `Scenario` (ABC) and must:
 
 1. **Define `VERSION`** as a class constant (increment on breaking changes)
 2. **Optionally declare `BASELINE_ATTACK_POLICY`** (defaults to `BaselineAttackPolicy.Enabled` — a baseline `PromptSendingAttack` is prepended and callers can opt out per run by setting `include_baseline=False` in the run params, see "Run Parameters" below):
-   - `BaselineAttackPolicy.Disabled` — baseline supported but off by default (e.g. `Jailbreak`, where templates dominate the run).
+   - `BaselineAttackPolicy.Disabled` — baseline supported but off by default (e.g. `garak.doctor`, where the harm-probe technique set dominates the run).
    - `BaselineAttackPolicy.Forbidden` — baseline is meaningless for this scenario's comparison axis (e.g. `AdversarialBenchmark`, which compares against gold-standard answers). Supplying `include_baseline=True` raises `ValueError`.
 3. **Pass `technique_class`, `default_technique`, and `default_dataset_config` to `super().__init__()`:**
 

doc/code/scenarios/0_attack_techniques.ipynb

@@ -59,7 +59,7 @@
     "  any scenario can use (the `role_play_*` variants, `many_shot`, `tap`, the `crescendo_*` variants, `red_teaming`,\n",
     "  `context_compliance`). Registered by default.\n",
     "- [`extra.py`](../../../pyrit/setup/initializers/techniques/extra.py) — opt-in techniques that are\n",
-    "  not part of the default set (`pair`, `violent_durian`).\n",
+    "  not part of the default set (`pair`, `violent_durian`, `skeleton_key`).\n",
     "- [`airt.py`](../../../pyrit/setup/initializers/techniques/airt.py) — source-owned techniques that\n",
     "  belong to a specific AIRT scenario. Unlike `core`/`extra`, these are imported directly by their\n",
     "  owning scenario and are *not* part of the default aggregation.\n",
@@ -112,6 +112,7 @@
       "  role_play_persuasion_written            PromptSendingAttack          yes single_turn, light, core\n",
       "         role_play_trivia_game            PromptSendingAttack          yes single_turn, light, core\n",
       "          role_play_video_game            PromptSendingAttack          yes single_turn, light, core\n",
+      "                  skeleton_key              SkeletonKeyAttack           no       single_turn, extra\n",
       "                           tap TreeOfAttacksWithPruningAttack          yes         multi_turn, core\n",
       "                violent_durian               RedTeamingAttack          yes        multi_turn, extra\n"
      ]

doc/code/scenarios/0_attack_techniques.py

@@ -58,7 +58,7 @@
 #   any scenario can use (the `role_play_*` variants, `many_shot`, `tap`, the `crescendo_*` variants, `red_teaming`,
 #   `context_compliance`). Registered by default.
 # - [`extra.py`](../../../pyrit/setup/initializers/techniques/extra.py) — opt-in techniques that are
-#   not part of the default set (`pair`, `violent_durian`).
+#   not part of the default set (`pair`, `violent_durian`, `skeleton_key`).
 # - [`airt.py`](../../../pyrit/setup/initializers/techniques/airt.py) — source-owned techniques that
 #   belong to a specific AIRT scenario. Unlike `core`/`extra`, these are imported directly by their
 #   owning scenario and are *not* part of the default aggregation.

doc/scanner/airt.ipynb

@@ -526,18 +526,28 @@
    "source": [
     "## Jailbreak\n",
     "\n",
-    "Tests target resilience against template-based jailbreak attacks using various prompt injection\n",
-    "templates.\n",
+    "Tests target resilience against jailbreak templates. A run crosses three selectors: the harmful\n",
+    "objectives (**dataset**, HarmBench), the **techniques** each jailbreak is delivered through, and\n",
+    "which **jailbreaks** to run. Two deliveries are on by default: `prompt_sending` renders the\n",
+    "objective inline into the template as a request converter (target-agnostic), and\n",
+    "`jailbreak_system_prompt` sets the template as a native system prompt with the objective sent as\n",
+    "the user turn (only for targets that natively support editable history + system prompts — it is\n",
+    "skipped for incapable targets). Registry techniques like `role_play_*`, `many_shot`, and `tap` are\n",
+    "opt-in. Results are grouped by jailbreak template, and a baseline (the un-jailbroken objective) is\n",
+    "included by default so complying with the bare objective is itself visible.\n",
     "\n",
     "```bash\n",
     "pyrit_scan airt.jailbreak \\\n",
-    "  --initializers target \\\n",
+    "  --initializers target load_default_datasets \\\n",
     "  --target openai_chat \\\n",
-    "  --techniques prompt_sending \\\n",
+    "  --dataset-names harmbench \\\n",
     "  --max-dataset-size 1\n",
     "```\n",
     "\n",
-    "**Available techniques:** ALL, SIMPLE, COMPLEX, PromptSending, ManyShot, SkeletonKey, RolePlay"
+    "**Available techniques:** ALL, DEFAULT (`prompt_sending` + `jailbreak_system_prompt`), plus registry\n",
+    "techniques (`role_play_*`, `many_shot`, `tap`, …). By default a small random sample of jailbreak\n",
+    "templates runs; pass `num_jailbreaks` (random count) or `jailbreak_names` (explicit) to widen or\n",
+    "pin the selection."
    ]
   },
   {
@@ -564,13 +574,14 @@
    "source": [
     "from pyrit.scenario.airt import Jailbreak, JailbreakTechnique\n",
     "\n",
-    "dataset_config = DatasetAttackConfiguration(dataset_names=[\"airt_harms\"], max_dataset_size=1)\n",
+    "dataset_config = DatasetAttackConfiguration(dataset_names=[\"harmbench\"], max_dataset_size=1)\n",
     "\n",
     "scenario = Jailbreak()\n",
     "scenario.set_params_from_args(  # type: ignore\n",
     "    args={\n",
     "        \"objective_target\": objective_target,\n",
-    "        \"scenario_techniques\": [JailbreakTechnique.PromptSending],\n",
+    "        \"scenario_techniques\": [JailbreakTechnique.DEFAULT],\n",
+    "        \"jailbreak_names\": [\"aim.yaml\"],\n",
     "        \"dataset_config\": dataset_config,\n",
     "    }\n",
     ")\n",

doc/scanner/airt.py

@@ -165,29 +165,40 @@
 # %% [markdown]
 # ## Jailbreak
 #
-# Tests target resilience against template-based jailbreak attacks using various prompt injection
-# templates.
+# Tests target resilience against jailbreak templates. A run crosses three selectors: the harmful
+# objectives (**dataset**, HarmBench), the **techniques** each jailbreak is delivered through, and
+# which **jailbreaks** to run. Two deliveries are on by default: `prompt_sending` renders the
+# objective inline into the template as a request converter (target-agnostic), and
+# `jailbreak_system_prompt` sets the template as a native system prompt with the objective sent as
+# the user turn (only for targets that natively support editable history + system prompts — it is
+# skipped for incapable targets). Registry techniques like `role_play_*`, `many_shot`, and `tap` are
+# opt-in. Results are grouped by jailbreak template, and a baseline (the un-jailbroken objective) is
+# included by default so complying with the bare objective is itself visible.
 #
 # ```bash
 # pyrit_scan airt.jailbreak \
-#   --initializers target \
+#   --initializers target load_default_datasets \
 #   --target openai_chat \
-#   --techniques prompt_sending \
+#   --dataset-names harmbench \
 #   --max-dataset-size 1
 # ```
 #
-# **Available techniques:** ALL, SIMPLE, COMPLEX, PromptSending, ManyShot, SkeletonKey, RolePlay
+# **Available techniques:** ALL, DEFAULT (`prompt_sending` + `jailbreak_system_prompt`), plus registry
+# techniques (`role_play_*`, `many_shot`, `tap`, …). By default a small random sample of jailbreak
+# templates runs; pass `num_jailbreaks` (random count) or `jailbreak_names` (explicit) to widen or
+# pin the selection.
 
 # %%
 from pyrit.scenario.airt import Jailbreak, JailbreakTechnique
 
-dataset_config = DatasetAttackConfiguration(dataset_names=["airt_harms"], max_dataset_size=1)
+dataset_config = DatasetAttackConfiguration(dataset_names=["harmbench"], max_dataset_size=1)
 
 scenario = Jailbreak()
 scenario.set_params_from_args(  # type: ignore
     args={
         "objective_target": objective_target,
-        "scenario_techniques": [JailbreakTechnique.PromptSending],
+        "scenario_techniques": [JailbreakTechnique.DEFAULT],
+        "jailbreak_names": ["aim.yaml"],
         "dataset_config": dataset_config,
     }
 )

pyrit/scenario/scenarios/airt/__init__.py

@@ -6,7 +6,7 @@
 from typing import Any
 
 from pyrit.scenario.scenarios.airt.cyber import Cyber, _build_cyber_technique
-from pyrit.scenario.scenarios.airt.jailbreak import Jailbreak, JailbreakTechnique
+from pyrit.scenario.scenarios.airt.jailbreak import Jailbreak, _build_jailbreak_technique
 from pyrit.scenario.scenarios.airt.leakage import Leakage, _build_leakage_technique
 from pyrit.scenario.scenarios.airt.psychosocial import Psychosocial, PsychosocialTechnique
 from pyrit.scenario.scenarios.airt.rapid_response import RapidResponse, _build_rapid_response_technique
@@ -29,6 +29,8 @@ def __getattr__(name: str) -> Any:
         return _build_leakage_technique()
     if name == "CyberTechnique":
         return _build_cyber_technique()
+    if name == "JailbreakTechnique":
+        return _build_jailbreak_technique()
     raise AttributeError(f"module {__name__!r} has no attribute {name!r}")