Skip to content

pii.md

Latest commit

 

History

History
178 lines (133 loc) · 6.43 KB

pii.md

File metadata and controls

178 lines (133 loc) · 6.43 KB
Raw

rag-pii — reference

PII detection and per-tenant policy enforcement for AgentContextOS. Step 1.7 ships two PIIDetector plugins, a PiiProcessor that applies the configured PiiPolicy from RequestContext, and pure text-rewriter helpers.

Overview

A PIIDetector (rag_core.spi.PIIDetector) takes text and returns ordered, non-overlapping PIISpans — one per detected entity. The detector knows what is PII; the PiiProcessor knows what to do about it. This split keeps detectors swappable without rewriting policy logic and lets the same processor run across regex, Presidio, or cloud detectors.

PIISpan is a Pydantic v2 frozen model in rag_core.types:

Field Type Notes
start int character offset, inclusive
end int character offset, exclusive
entity_type str engine-defined (EMAIL, PERSON, …)
score float 0.0–1.0 confidence
replacement str what to substitute on redact (<EMAIL> etc.)

PIISpan.__init__ validates end >= start and 0 <= score <= 1.

Detectors

RegexPIIDetector

Pure-Python, always available. Catches EMAIL, PHONE, SSN, CREDIT_CARD (Luhn-checked), IP_ADDRESS. Confidence scores are type-specific (EMAIL=0.95, PHONE=0.7 — phone has the highest false-positive rate).

from rag_pii import RegexPIIDetector

detector = RegexPIIDetector()
spans = await detector.detect(ctx, "ping alice@example.com today")
redacted = await detector.redact(ctx, "...")

PresidioPIIDetector

Behind the [presidio] extra: pip install "rag-pii[presidio]" plus the spaCy English model (python -m spacy download en_core_web_sm). Adds PERSON, LOCATION, DATE_TIME, US_DRIVER_LICENSE, EMAIL_ADDRESS, and more (full list in Presidio's docs).

from rag_pii import PresidioPIIDetector

detector = PresidioPIIDetector(language="en", min_score=0.5)

min_score lets the detector pre-filter — useful for production deployments that want a strict floor before the processor's policy filter runs.

PiiProcessor

from rag_pii import PiiProcessor, RegexPIIDetector

processor = PiiProcessor(detector=RegexPIIDetector())
filtered_chunks, outcomes = await processor.process(ctx, chunks)

Reads ctx.pii_policy (a PiiPolicy with action, entities, min_score) and applies it per chunk:

Action Behaviour
redact Spans replaced with <TYPE> markers (default)
mask Spans replaced with * of equal length (preserves layout)
block Chunk dropped from output list

Returns (filtered_chunks, outcomes) where outcomes is a list of PiiOutcome(chunk_id, action, entity_types, span_count) describing what was done to each affected chunk — useful for trace + audit log without re-running the detector.

Every affected chunk emits one pii.detected structured event per entity type via rag-observability. Events carry only metadata (type, chunk_id, count) — never raw matched text — so log sinks are safe even when redaction is disabled.

PiiPolicyEngine — egress enforcement (Step 6.5)

PiiProcessor runs at ingest. PiiPolicyEngine is its egress sibling: a PolicyEngine decorator (like QuotaPolicyEngine / AclPolicyEngine) that answers the egress_text decision over text about to leave the system, applying the per-tenant ctx.pii_policy action.

from rag_pii import PiiPolicyEngine, RegexPIIDetector
from rag_policy import NoopPolicyEngine

engine = PiiPolicyEngine(inner=NoopPolicyEngine(), detector=RegexPIIDetector())
# evaluate(ctx, PolicyDecision.egress_text, subject) where subject is a str
# (the generated answer) or a list[Chunk] (the retrieved context).
pii_policy.action Result Notes
allow allow passes through unchanged
redact transform spans → <TYPE> markers (str or each chunk's content)
mask transform spans → * of equal length
block deny text withheld; the caller drops the answer / context

The gateway already calls policy_engine.evaluate(ctx, egress_text, …) at every generation boundary — the retrieved context (list[Chunk]) on /v1/query?generate / /v1/chat/completions / gRPC Converse, and the final answer (str) on the agent path — so enabling the engine enforces PII at all of them without any route change. Spans are filtered by the tenant's min_score

  • entities before an action is taken; a clean scan is a no-op. A block emits a PII-free pii.egress_blocked event (entity types + counts only); a redact/mask emits pii.detected. Non-egress_text decisions, filter_pushdown, and health delegate to the inner engine, so it composes with the ACL / quota PDP.

Enable via cfg.pii.enabled (off by default; the gateway injects RegexPIIDetector unless a pii_detector is supplied).

Rewriters

from rag_pii import redact_spans, mask_spans

Pure functions used by the detectors and the processor; exposed for custom enforcement flows.

  • redact_spans(text, spans) -> str — splice replacements at span positions
  • mask_spans(text, spans, mask_char="*") -> str — replace with same-length mask characters

Both assume spans are ordered + non-overlapping (the PIIDetector SPI contract requires this) and skip out-of-range spans defensively.

CLI

ragctl pii path/to/doc.md                              # regex + redact (default)
ragctl pii doc.md --action mask
ragctl pii doc.md --action block
ragctl pii doc.md --detector presidio --action redact
ragctl pii doc.md -n 0                                 # skip per-chunk preview

Pipes parse → chunk → enrich → PII processor. Prints per-type tally, in/out chunk counts (affected by block), and a chunk preview.

Errors

All PII failures wrap to rag_core.errors.IngestionError:

  • Detector internals crash (Presidio engine, regex compile error)
  • Empty content edge cases
  • Missing extras (clear install hint in the message)

Extension points

Implement a custom detector:

  1. Subclass rag_core.spi.PIIDetector.
  2. Implement detect(ctx, text) -> list[PIISpan] returning ordered, non-overlapping spans.
  3. Implement redact(ctx, text) -> str (typically delegates to redact_spans() after detect()).
  4. Map third-party exceptions to IngestionError.
  5. Add a health() returning True when the engine is reachable.

The PiiProcessor works with any PIIDetector — custom detectors plug in for free.