[DRAFT] FEAT Add Turkish Conversation Prompt-Injection dataset loader

[3nesdeniz]

Description

Closes #2171.

This PR adds a first-class PyRIT loader for the public Turkish Conversation Prompt-Injection Dataset (CC BY 4.0). The dataset contains 750 synthetic, curated Turkish examples across 10 prompt-injection families, including 150 benign boundary cases paired with related attacks. It supports both attack detection and over-refusal testing in Turkish LLM security workflows.

The loader:

• defaults to attack examples from all published splits;
• provides typed filters for attack, benign, or all labels, split selection, and one or more attack families;
• pins the immutable v1.0.1 Hugging Face revision for reproducible loading;
• validates the published schema, label-dependent fields, source contexts, split metadata, unique IDs, and complete same-context pairs before filtering;
• preserves source context, pair ID, source type, split, label, attack family, dataset version, and Hugging Face revision as row-level provenance;
• keeps attack families in metadata rather than treating delivery techniques as PyRIT harm categories;
• keeps paired records as independent evaluation cases while preserving pair_id in metadata;
• supports the existing Hugging Face token and cache behavior; and
• raises PyRIT's standard explicit error when a valid filter produces an empty dataset.

The loader is registered with SeedDatasetProvider, documented in the dataset-loading guide, and linked to a bibliography entry. Attack examples contain adversarial instructions and should only be used in controlled, authorized security testing.

Tests and documentation

• Focused loader and provider validation: 218 passed
• Full unit suite after hardening: 10,638 passed, 119 skipped
• Targeted pre-commit checks passed, including Ruff, type checking, notebook checks, documentation validation, and secret detection
• Strict Jupytext and targeted strict documentation builds passed
• Live pinned-release validation confirmed 750 rows, the 600/150 label distribution, 150 complete pairs, all 10 attack families, 17 source contexts, and provider discovery
• The live v1.0.1 schema was checked directly; it does not publish a row-level language field, so no synthetic metadata field is introduced

AI assistance was used for implementation drafting and test planning. I reviewed the resulting code and documentation, validated them against the live dataset, and ran all checks listed above.

[3nesdeniz]

@microsoft-github-policy-service agree

[romanlutz]

Is there a paper associated with this? How were they collected or curated? Do they actually work on some model?

[3nesdeniz]

Thanks for the questions. There is no associated paper or preprint at this stage. The citation refers to the versioned Hugging Face dataset release itself.

The dataset was created through a paired curation process. I defined ten prompt-injection attack families and prepared 150 attack examples. For every attack, I created a matched legitimate request that remains close in topic, context, and vocabulary but does not attempt to override instructions or cross a trust boundary. These boundary pairs were combined with 450 additional legitimate requests, resulting in 600 benign examples and 150 attacks.

This structure is intentional. A detector should not classify a request as malicious merely because it contains terms such as "system prompt," "admin," or "API key." It must distinguish between legitimate user intent and an actual attempt to override authority or extract protected information. The matched benign examples serve as hard negatives and are intended to support false-positive analysis and reduction.

The attack examples were also executed in AltayDuel, AltaySec's continuously operating AI-vs-AI prompt-injection arena. AltayDuel runs Red and Blue agents across multiple model providers and records the complete conversation transcript, model and provider metadata, and judge verdicts. The attack examples included in this release produced successful prompt-injection outcomes under the arena's documented evaluation criteria.

These results should not be interpreted as universal bypasses against every model or configuration. Prompt-injection success remains dependent on the target model, system prompt, available tools, and surrounding application context.

The dataset repository also includes deterministic pair-preserving splits and validation for schema consistency, duplicate rows, pair integrity, split leakage, attack-family balance, and common sensitive-data patterns.

[romanlutz]

Understood. I'll examine it in more detail after our next release is out. That may take at least the next week. Thanks for your patience.