Skip to content

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#2175

Open
3nesdeniz wants to merge 5 commits into
microsoft:mainfrom
3nesdeniz:feat/agentic-prompt-injection-boundary-dataset
Open

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader#2175
3nesdeniz wants to merge 5 commits into
microsoft:mainfrom
3nesdeniz:feat/agentic-prompt-injection-boundary-dataset

Conversation

@3nesdeniz

@3nesdeniz 3nesdeniz commented Jul 13, 2026

Copy link
Copy Markdown

Description

Closes #2174.

This PR adds a remote loader for the public CC BY 4.0 Agentic Prompt-Injection Boundary Pairs dataset:

  • 1,200 English records organized as 600 controlled benign/attack pairs;
  • 50 scenario-isolated enterprise and agentic workflows;
  • 12 prompt-injection and trust-boundary attack families;
  • train, validation, test, or combined loading;
  • attack-only default plus benign and full-pair modes;
  • typed filters for label, split, attack family, and source context;
  • pair, scenario, source, boundary, expected-action, version, and revision provenance on each seed.

The loader maps each row to a literal SeedPrompt and pins the immutable v1.0.0 Hugging Face revision. Before filtering, it validates required fields, label-dependent values, allowed families and source contexts, split metadata, duplicate IDs, and complete pair integrity.

Family filtering uses pair_family so full-pair mode preserves both sides of a controlled pair. Agent-security decision labels and attack families remain in metadata rather than being misclassified as PyRIT harm categories. Pair members remain independent evaluation cases; pair_id preserves the relationship without collapsing two prompts into one SeedGroup.

Source and release pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs

I am the dataset author and maintain the tagged source release. The dataset is synthetic, manually reviewed, and contains no customer or private data.

Tests and documentation

  • Focused loader and dataset-provider validation: 217 passed
  • Full dataset unit-test package: 1,171 passed
  • Full unit suite on current main: 10,720 passed, 119 skipped
  • Targeted pre-commit checks passed, including Ruff, changed-file type checking, notebook sanitation, documentation validation, secret detection, and JSON checks
  • Strict Jupytext validation passed; the paired source and notebook discovery output are synchronized
  • Live pinned-release loading confirmed 600 default attack seeds and 240 test rows forming 120 complete pairs
  • The public release was verified as 1,200 rows, 600 pairs, 50 scenarios, with matching JSONL/Parquet artifacts and checksums
  • One real matched pair from each of the 12 attack families was manually inspected

The loader is registered in dataset discovery and documented in the loading guide, notebook, and project bibliography.

@3nesdeniz 3nesdeniz changed the title [DRAFT] FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader Jul 16, 2026
@3nesdeniz 3nesdeniz marked this pull request as ready for review July 16, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader

1 participant