Skip to content

provenance.md

Latest commit

 

History

History
142 lines (117 loc) · 6.1 KB

provenance.md

File metadata and controls

142 lines (117 loc) · 6.1 KB
Raw

Provenance & per-query tracing (rag-provenance, Step 5.1)

Overview

Every query can leave a tamper-evident record of what produced its answer — the query, the routing decision, the evidence (chunk/document locators + scores), and the model — captured at the gateway, signed with HMAC-SHA256, and retrievable together with the query's captured span tree at GET /v1/query/{id}/trace.

It is on by default but behaviour-neutral: recording never rejects or alters a response, and a store failure degrades to a logged event rather than an error. Raw query/answer text is not persisted unless the operator opts in — only SHA-256 hashes.

Usage

Retrieve a query's trace

# 1. run a query (note the request_id in the response)
curl -s localhost:8000/v1/query \
  -H 'X-Tenant-Id: acme' \
  -d '{"tenant_id":"acme","principal_id":"alice","query":"who signed the lease?"}'
# => { "request_id": "5f3a…", ... }

# 2. fetch its provenance + spans (header identity must match the tenant)
curl -s localhost:8000/v1/query/5f3a…/trace \
  -H 'X-Tenant-Id: acme' -H 'X-Principal-Id: alice'

Response (QueryTraceResponse):

{
  "request_id": "5f3a…",
  "trace_id": "0f9f…",
  "found": true,
  "provenance": {
    "record": {
      "request_id": "5f3a…", "trace_id": "0f9f…", "tenant_id": "acme",
      "surface": "rest.query",
      "query_hash": "a5be…",          // sha256; query_text is null by default
      "routing_shape": "semantic", "backends_used": ["vector", "keyword"],
      "citations": [{"chunk_id":"c1","document_id":"d1","source_uri":"…","score":0.92}],
      "answer_id": "…", "answer_hash": "…", "model": "…",
      "schema_version": "1.0", "created_at": "2026-06-04T…"
    },
    "signature": {"algorithm":"HMAC-SHA256","timestamp":1700000000,"value":"…"}
  },
  "verification": {"signed": true, "verified": true, "reason": "ok"},
  "spans": [
    {"name":"gateway.query","span_id":"…","duration_ms":4.1,
     "attributes":{"rag.schema_version":"1.0","rag.request_id":"5f3a…", …}},
    {"name":"retrieve.route", …}
  ]
}

Status codes: 200 found · 404 unknown id / other tenant / provenance disabled ({"error":{"code":"provenance_not_found", …}}) · 401 missing auth.

CLI smoke test (no server, no secrets)

ragctl provenance "who approved invoice 42?" -s topsecret --capture-text

Builds → signs → stores → retrieves → verifies a record against noop SPIs and prints it plus the captured spans, then tampers with the record to show verification flipping to False.

Configuration (rag.yaml)

provenance:
  enabled: true                 # on by default; behaviour-neutral + degrade-open
  signing_secret: "${PROVENANCE_HMAC_SECRET}"   # empty => records stored unsigned
  capture_query_text: false     # privacy: false stores only the query hash
  capture_answer_text: false
  capture_spans: true           # install the TraceCollector for the trace endpoint
  max_records: 10000            # in-memory store bound (single process)
  max_traces: 1000
  max_spans_per_trace: 256

Programmatic

from rag_provenance import ProvenanceRecorder, ProvenanceSigner
from rag_core.spi.noop import NoopProvenanceStore

recorder = ProvenanceRecorder(NoopProvenanceStore(), signer=ProvenanceSigner("secret"))
signed = await recorder.capture(
    ctx, surface="rest.query", query=text,
    decision=decision, citations=citations, answer=answer,
    timings_ms={"total_ms": 12.0}, corpus_ids=corpus_ids,
)                                # None if recording degraded
assert recorder.verify(signed).verified

Public surface

Symbol Package What
ProvenanceRecorder rag_provenance build / sign / capture (degrade-open) / verify
ProvenanceSigner rag_provenance HMAC-SHA256 sign / verify; enabled
ProvenanceRecord · ProvenanceCitation rag_core.types the record + an evidence locator
ProvenanceSignature · SignedProvenanceRecord rag_core.types detached signature + the bundle
ProvenanceVerification rag_core.types signed / verified / reason
SpanRecord rag_core.types one captured span (name, ids, timing, status, attributes)
ProvenanceStore · NoopProvenanceStore rag_core.spi / .noop put / get (ctx-first, tenant-scoped, bounded)
QueryTraceResponse rag_core.gateway_types the /v1/query/{id}/trace body
TraceCollector · install_trace_collector rag_observability span processor + read side
ProvenanceConfig rag_config the provenance: section

Internals

  • Signing = HMAC-SHA256 over f"{timestamp}." + record.model_dump_json(), hex digest in signature.value — the same scheme as rag_webhooks.signing. Verification re-serialises the stored record; any field mutation flips verified to False (reason="mismatch").
  • Capture order: build (hash query/answer, derive backends_used from the routing decision, fill schema_version) → sign (if a secret is set) → store.put → emit provenance.recorded. Wrapped in a provenance.record span. Any exception → provenance.record_degraded + return None.
  • Span grouping: the TraceCollector buckets finished spans by the rag.trace_id attribute (on every span via span_from_trace_context). The trace endpoint resolves that id from the signed record (looked up by request_id).

Extension points

  • Cross-worker storage: implement ProvenanceStore over Redis/Postgres and inject it as provenance_store (mirrors RedisQuotaStore). The in-memory default is single-process.
  • Real signing/KMS: ProvenanceSigner takes a plain secret; a key_id rides on every signature for rotation bookkeeping. Swap in a KMS-backed signer behind the same sign / verify shape.
  • Custom evidence: the recorder's build assembles ProvenanceCitations from Citations; override at the call site to capture additional locators.

See ADR-0026 and architecture/per-query-tracing.md.