FEAT expand TargetCapabilities by hannahwestra25 · Pull Request #1464 · Azure/PyRIT

hannahwestra25 · 2026-03-13T20:06:05Z

Description

This PR builds on #1433 to expand the TargetCapabilities class and consolidate the logic in Targets to use the TargetCapabilities class rather than misc variables. This is the first of at least 2 more PRs which will allow users to query target capabilities and add validation to attacks, converters, scorers, etc which have requirements for targets.

TargetCapabilities dataclass — expanded fields:

supports_multi_turn (existing)
supports_multi_message_pieces — rejects messages with >1 piece when False
supports_json_response — whether JSON response format is supported
input_modalities — allowed input data types (text, image_path, audio_path, …)
output_modalities — produced output data types

Added assert_satisfies() to validate one TargetCapabilities against another. This will be useful later when validating whether a target satisfies the requirements of an attack / scorer / converter

PromptTarget._validate_request() — converted from abstract to a concrete base
implementation that auto-enforces capabilities:

Rejects multi-piece messages when supports_multi_message_pieces=False
Rejects unsupported converted_value_data_type against input_modalities
Rejects follow-up turns when supports_multi_turn=False

All ad-hoc inline validation in individual targets was deleted.
Renamed constructor param capabilities → custom_capabilities.
Added is_json_response_supported() delegating to capabilities.supports_json_response.

Per-target _DEFAULT_CAPABILITIES declarations added/updated:

Target	Notable capabilities
`OpenAIChatTarget`	multi-turn, JSON, text/image/audio in, text/audio out
`OpenAICompletionTarget`	`supports_multi_message_pieces=False`
`OpenAITTSTarget`	no multi-turn, single-piece, audio out
`OpenAIImageTarget`	no multi-turn, text/image in, image out
`OpenAIVideoTarget`	no multi-turn, text/image in, video out
`RealtimeTarget`	multi-turn, single-piece, text/audio in+out
`PlaywrightTarget`	multi-turn, text/image in
`PlaywrightCopilotTarget`	multi-turn, text/image in+out
`WebSocketCopilotTarget`	multi-turn, text/image in
`PromptShieldTarget`	single-piece
`HuggingFaceChatTarget`	multi-turn, single-piece
`CrucibleTarget`	single-piece

Tests and Documentation

New test_target_capabilities.py covering modality declarations, assert_satisfies, and per-target defaults
test_supports_multi_turn.py extended with constructor override tests
All pytest.raises error message patterns updated to the new unified format
Mock memory calls updated: get_conversation → get_message_pieces
patch_central_database fixture added to HTTP target tests that were missing it

ValbuenaVC · 2026-03-13T21:29:17Z

pyrit/prompt_target/common/target_capabilities.py

    Each target class defines default capabilities via the _DEFAULT_CAPABILITIES
    class attribute. Users can override individual capabilities per instance
    through constructor parameters, which is useful for targets whose
    capabilities depend on deployment configuration (e.g., Playwright, HTTP).


Can we document whether or not _DEFAULT_CAPABILITIES is required for all targets? Since we define an empty one in the PromptTarget base class, this might be a nit since users don't necessarily have to add it to their own targets, but we should be explicit so users adding new targets can do so easily.

Also a nit, I'm also not seeing any tests to ensure all targets have this attribute. We don't necessarily need a regression test but it might be worth adding if we expect most or all targets here to use it

ValbuenaVC · 2026-03-13T21:30:48Z

tests/unit/target/test_video_target.py


    mock_memory = MagicMock()
-    mock_memory.get_conversation.return_value = sample_conversations
+    mock_memory.get_message_pieces.return_value = sample_conversations


Why was this changed?

ValbuenaVC · 2026-03-13T21:32:27Z

pyrit/prompt_target/common/target_capabilities.py

+    input_modalities: list[PromptDataType] = field(default_factory=lambda: ["text"])
+
+    # The output modalities supported by the target (e.g., "text", "image").
+    output_modalities: list[PromptDataType] = field(default_factory=lambda: ["text"])


Nit: What do you think about making these into sets? It would be functionally identical but I'm curious if we want to preserve the ordering

rlundeen2 · 2026-03-13T22:33:48Z

pyrit/prompt_target/common/target_capabilities.py

+    supports_multi_message_pieces: bool = True
+
+    # Whether the target natively supports JSON output (e.g., via a "json" response format).
+    supports_json_response: bool = False


You likely want to split this to json_schema_support and json_output_support because they're different params. In one you include the schema, in the other you put json=True

rlundeen2 · 2026-03-13T22:35:02Z

pyrit/prompt_target/common/target_capabilities.py

+    # Whether the target natively supports JSON output (e.g., via a "json" response format).
+    supports_json_response: bool = False
+
+    # The input modalities supported by the target (e.g., "text", "image").


We also likely want editable_history; This let's us set the system history that are not actually responses from the target. We use PromptChatTarget for this currently.

rlundeen2 · 2026-03-13T22:36:05Z

pyrit/prompt_target/common/prompt_chat_target.py

-import abc
 from typing import Optional

 from pyrit.identifiers import ComponentIdentifier


We can probably deprecate PromptChatTarget and remove it from the inheritence chain, since we should check via prompt capabilities now

note this may be part of a future PR

rlundeen2 · 2026-03-13T22:37:24Z

pyrit/prompt_target/common/prompt_chat_target.py

+            custom_capabilities=custom_capabilities,
        )

    def set_system_prompt(


We should probably get rid of this also. set_system_prompt is error prone and you can always do it with prepended_conversation

note this may be part of a future PR. We also need to think through how we set the system prompt for things like RealTimeTarget since they are multi-turn, you can set the system prompt, but you can't edit the history.

rlundeen2 · 2026-03-13T22:37:53Z

pyrit/prompt_target/common/prompt_target.py

        self._underlying_model = underlying_model
-        self._capabilities = capabilities if capabilities is not None else type(self)._DEFAULT_CAPABILITIES
+        self._capabilities = (
+            custom_capabilities if custom_capabilities is not None else type(self)._DEFAULT_CAPABILITIES


good, I like this

romanlutz · 2026-03-13T22:39:25Z

pyrit/prompt_target/common/target_capabilities.py

+    # The input modalities supported by the target (e.g., "text", "image").
+    input_modalities: list[PromptDataType] = field(default_factory=lambda: ["text"])
+
+    # The output modalities supported by the target (e.g., "text", "image").
+    output_modalities: list[PromptDataType] = field(default_factory=lambda: ["text"])


I worry about this a little. You have multi-piece and input modalities, but just because an endpoint supports text and image and video and multiple pieces doesn't mean it supports all combinations of those, right?

It might just accept text+image and text+video, but not all three together, or not image+video.

A solution to this would be to drop the multi-piece support field, and instead make input and output modalities of type set[set[PromptDataType]] (set because the order is irrelevant and we can check membership in constant time (although complexity is hardly an issue here). I think @fitzpr's PR #1383 has this as sets of sets if you want an idea.

Wdyt? Did you consider this and decide it's not good for some reason? I might be missing some cases because I haven't spent as much time on it.

jsong468 · 2026-03-13T22:50:44Z

pyrit/prompt_target/common/prompt_target.py

        self._model_name = model_name
        self._underlying_model = underlying_model
-        self._capabilities = capabilities if capabilities is not None else type(self)._DEFAULT_CAPABILITIES
+        self._capabilities = (


self._capabilities is defined here but the later changes reference self.capabilities?

rlundeen2 · 2026-03-13T22:54:26Z

pyrit/prompt_target/common/prompt_target.py


        return ComponentIdentifier.of(self, params=all_params, children=children)

+    def is_json_response_supported(self) -> bool:


I think we should remove the methods like this (and supports_multi_turn, etc, and just have people validate using the capabilities property.

jsong468 · 2026-03-13T22:57:18Z

pyrit/prompt_target/common/prompt_target.py

+                    f"This target supports only the following data types: {supported_types}. Received: {piece_type}."
+                )
+
+        if not self.supports_multi_turn:


self.capabilities.supports_multi_turn? Right now, we have a convenience property for this but not some of the others. Since we have a bunch now, we could just get rid of it?

rlundeen2 · 2026-03-13T22:58:37Z

pyrit/prompt_target/common/target_capabilities.py

    supports_multi_turn: bool = False
+
+    # Whether the target natively supports multiple message pieces in a single request.
+    supports_multi_message_pieces: bool = True


I might default to the least capable; supports_multi_message_pieces = False

rlundeen2 · 2026-03-13T22:59:45Z

pyrit/prompt_target/openai/openai_chat_target.py


    """

+    _DEFAULT_CAPABILITIES: TargetCapabilities = TargetCapabilities(


Similarly, without knowing the endpoint, it might make sense to make this restrictive. It depends on the type of openai endpoint which modalities and json support it has.

rlundeen2 · 2026-03-13T23:01:20Z

pyrit/prompt_target/common/prompt_target.py

                from the actual model. If not provided, `model_name` will be used for the identifier.
                Defaults to None.
-            capabilities (TargetCapabilities, Optional): Override the default capabilities for
+            custom_capabilities (TargetCapabilities, Optional): Override the default capabilities for


I wonder if we could have a mapping of known default capabilities here, or potentially retrieved from a list in target_capabilites class.

if underlying_model == "gpt-5.1": _default = X elif ...

We also talked about a method to discover these which I think will be useful. But the defaults we know could go a long way early on :)

rlundeen2 · 2026-03-13T23:04:20Z

pyrit/prompt_target/openai/openai_image_target.py

-    _DEFAULT_CAPABILITIES: TargetCapabilities = TargetCapabilities(supports_multi_turn=False)
+    _DEFAULT_CAPABILITIES: TargetCapabilities = TargetCapabilities(
+        supports_multi_turn=False,
+        input_modalities=["text", "image_path"],


Depends, many only support input_modality text. If we go the route of most restrictive, that'd be the default.

rlundeen2 · 2026-03-13T23:04:41Z

pyrit/prompt_target/openai/openai_image_target.py

        Initialize the image target with specified parameters.

        Args:
-            model_name (str, Optional): The name of the model (or deployment name in Azure).


Should we keep the doc helpers here?

pyrit/prompt_target/openai/openai_image_target.py

rlundeen2 · 2026-03-13T23:06:22Z

pyrit/prompt_target/openai/openai_response_target.py


        Args:
            custom_functions: Mapping of user-defined function names (e.g., "my_func").
-            model_name (str, Optional): The name of the model (or deployment name in Azure).


I think re-adding these doc strings?

rlundeen2 · 2026-03-13T23:08:27Z

pyrit/prompt_target/playwright_target.py

    SUPPORTED_DATA_TYPES = {"text", "image_path"}
-    _DEFAULT_CAPABILITIES: TargetCapabilities = TargetCapabilities(supports_multi_turn=True)
+    _DEFAULT_CAPABILITIES: TargetCapabilities = TargetCapabilities(
+        supports_multi_turn=True,


maybe only minimal since it's generic?

rlundeen2 · 2026-03-13T23:08:38Z

pyrit/prompt_target/playwright_target.py

        interaction_func: InteractionFunction,
        page: "Page",
        max_requests_per_minute: Optional[int] = None,
-        capabilities: Optional[TargetCapabilities] = None,


custom_capabilities here?

jsong468 · 2026-03-14T00:00:32Z

pyrit/prompt_target/openai/openai_chat_target.py

        is_json_supported: bool = True,
        audio_response_config: Optional[OpenAIChatAudioConfig] = None,
        extra_body_parameters: Optional[dict[str, Any]] = None,
+        custom_capabilities: Optional[TargetCapabilities] = None,


why don't we expose this param in other prompt target child classes?

jsong468 · 2026-03-14T00:01:09Z

pyrit/prompt_target/openai/openai_tts_target.py

 class OpenAITTSTarget(OpenAITarget):
    """A prompt target for OpenAI Text-to-Speech (TTS) endpoints."""

-    _DEFAULT_CAPABILITIES: TargetCapabilities = TargetCapabilities(supports_multi_turn=False)


does this support multi turn now?

hannahwestra25 added 4 commits March 13, 2026 14:57

build out TargetCapabilities class and consolidate old logic

8900ed5

pre-commit

9a39cc7

merge

a687aa1

add raises

4537a8b

ValbuenaVC reviewed Mar 13, 2026

View reviewed changes

rlundeen2 reviewed Mar 13, 2026

View reviewed changes

romanlutz reviewed Mar 13, 2026

View reviewed changes

jsong468 reviewed Mar 13, 2026

View reviewed changes

rlundeen2 reviewed Mar 13, 2026

View reviewed changes

jsong468 reviewed Mar 13, 2026

View reviewed changes

rlundeen2 reviewed Mar 13, 2026

View reviewed changes

pyrit/prompt_target/openai/openai_image_target.py Show resolved Hide resolved

rlundeen2 reviewed Mar 13, 2026

View reviewed changes

jsong468 reviewed Mar 14, 2026

View reviewed changes


		return ComponentIdentifier.of(self, params=all_params, children=children)

		def is_json_response_supported(self) -> bool:


		"""

		_DEFAULT_CAPABILITIES: TargetCapabilities = TargetCapabilities(

Conversation

hannahwestra25 commented Mar 13, 2026

Description

Tests and Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsong468 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlundeen2 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsong468 Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rlundeen2 Mar 13, 2026 •

edited

Loading

rlundeen2 Mar 13, 2026 •

edited

Loading

rlundeen2 Mar 13, 2026 •

edited

Loading

rlundeen2 Mar 13, 2026 •

edited

Loading

jsong468 Mar 13, 2026 •

edited

Loading

rlundeen2 Mar 13, 2026 •

edited

Loading

rlundeen2 Mar 13, 2026 •

edited

Loading

jsong468 Mar 14, 2026 •

edited

Loading