Skip to content

performance.md

Latest commit

 

History

History
146 lines (103 loc) · 10.2 KB

performance.md

File metadata and controls

146 lines (103 loc) · 10.2 KB
Raw

Performance — discipline, budgets, and reviewer checklist

Overview

This document captures the cross-cutting performance disciplines every contributor follows. It exists because performance at million-document scale is an architectural concern, not a tuning pass — the choices that determine whether you can hit p99 budgets are made at PR-review time, not during the Phase 4 latency sweep.

Three sections: hot-path discipline (how code is written), per-SPI budgets (what's measured), reviewer checklist (what's enforced at PR time).

Hot-path discipline

Pydantic at SPI boundaries only

Pydantic v2 validation is fast (~µs per model) but it runs every time. In a hot loop running 10k times per query, that's tens of milliseconds of pure overhead. Convention:

Layer Type Validation
Network boundary (gateway HTTP/gRPC) Pydantic Validate once
Public SPI surface Pydantic Validate once
Inside a single SPI call (loops) dataclass, msgspec, or Model.model_construct() None
Process-internal cross-stage envelopes (RequestContext, QueryPlan) Pydantic Validate at construction; treat as trusted

When trusted internal construction is needed for a Pydantic model, use Model.model_construct(**fields) — this skips validation entirely. Reserve for code paths where inputs are known-good (e.g., reconstructing a RequestContext from a cache).

Concrete examples of "hot loop" in this codebase:

Site Hot or cold Rule
Gateway request handler (per request) Cold Validate RequestContext once.
VectorRetrievalBackend.retrieve_ids returning 200 ChunkRefs Hot Construct ChunkRef with ChunkRef.model_construct(...) inside the comprehension.
Embedder.bulk_embed mapping provider response → 100 Embeddings Hot Embedding.model_construct(...) per item; the provider response shape is already validated by the SDK.
Pipeline stage function called once per item Caller-determined If the item comes from another SPI it's trusted — skip validation; if from connector.read (raw bytes) validate once at the boundary.
AuditWriter.write per event Hot under spikes The AuditEvent is built by trusted code — model_construct is fine.

Avoid per-call getLogger

logging.getLogger(__name__) is fast on modern Python but still has overhead in tight loops. Always store at module level:

from rag_observability.logging import get_logger
_log = get_logger(__name__)        # module level — once

async def hot_loop():
    for item in items:
        _log.debug("processing", extra={"id": item.id})

(This is also enforced by the RAG001 pre-commit hook for logging.getLogger.)

Async all the way

Any sync I/O call in the request path will tank p99 under load. Default to async clients for every backend. If a library only ships sync, wrap with asyncio.to_thread() at the SPI boundary, not in the hot path.

Bounded queues, never unbounded

asyncio.Queue() defaults to unbounded. That's a foot-gun: a slow consumer + fast producer = OOM. Always pass maxsize=. The Pipeline primitive (Step 1.1d) enforces this.

Don't hydrate what you'll throw away

Retrieve ChunkRef (IDs + scores), rerank, then hydrate only the survivors. See Step 1.1b for the SPI shape.

Batch concurrent provider calls

Use the Batcher middleware (Step 1.1d) for any SPI backed by a billed external API (embedders, rerankers, LLMs). Coalesces N concurrent calls into 1 provider call. Cuts cost ~10× and increases throughput ~3×.

Per-SPI p99 budgets

Published in tests/contract/budgets.py (real as of Step 4.6 — the per-SPI table below as structured data + the gateway budgets). A real backend implementation is held to its row by its own perf harness; the gateway-overhead row is actively enforced by the Step 4.6 latency gate (see below).

SPI Method p99 budget (dev profile) p99 budget (SaaS profile)
Embedder embed (single) 50 ms 30 ms
Embedder bulk_embed (batch 100) 500 ms 200 ms
VectorRetrievalBackend retrieve_ids (top 100) 50 ms 20 ms
VectorRetrievalBackend hydrate (50 chunks) 30 ms 15 ms
KeywordRetrievalBackend retrieve_ids 30 ms 15 ms
Reranker fast_rerank (200 → 50) 10 ms 5 ms
Reranker precise_rerank (50 → 10) 200 ms 100 ms
PolicyEngine (noop) evaluate 100 µs 100 µs
PolicyEngine (filter_pushdown) — 500 µs 500 µs
EmbeddingCache get 2 ms 1 ms
RetrievalCache get 2 ms 1 ms
AnswerCache get 2 ms 1 ms
AuditWriter write (async) 5 ms 2 ms

Gateway end-to-end (cache miss): p99 ≤ 500 ms dev, ≤ 250 ms SaaS — the Phase 2 exit gate. This is with real backends and is backend-bound, so it is a documented operational target, not a CI gate.

Gateway overhead gate (Step 4.6): p99 ≤ 30 ms. Distinct from the end-to-end target above: measured in-process against the noop backends, so it isolates the gateway's own per-request cost (framework + middleware + pipeline + (de)serialisation). It runs as a dedicated single-runner CI job (tests/perf/, the perf marker), reading the server-side gateway.request_duration_ms metric. See latency.md, reference/perf.md, and ADR-0025.

When a budget is missed, the conformance suite fails. The fix is either (a) tune the implementation or (b) raise the budget with an ADR explaining why.

Async telemetry path

OTel exporter back-pressure cannot block the request path. Shipped as rag_observability.AsyncTelemetrySink (Step 1.1e):

  • Bounded buffer: default 10,000 records. Configurable via max_buffer=.
  • Non-blocking submit(): drops on overflow, never raises and never awaits the queue.
  • Per-kind drop counter: sink.dropped_total("logs" | "spans" | "stage_events" | ...). Surface as the telemetry.dropped_total{kind} metric for alerting.
  • Per-kind exporter-error counter: sink.export_errors_total("..."). Exporter exceptions are swallowed (the drainer stays alive) but counted; surface as telemetry.export_errors_total{kind}.
  • Cooperative shutdown: await sink.stop() flushes pending records up to drain_timeout_s (default 5 s) before cancelling the drainer.
from rag_observability import AsyncTelemetrySink

async def export(rec):  # OTel exporter, log shipper, etc.
    await collector.send(rec)

sink = AsyncTelemetrySink(export, max_buffer=10_000)
sink.start()

sink.submit({"span": "vector.retrieve", "ms": 12}, kind="spans")   # non-blocking
sink.submit({"event": "ingest.completed"}, kind="stage_events")

# at shutdown
await sink.stop()

Tested via tests/observability/test_async_sink.py (9 tests covering drain ordering, drop accounting under simulated back-pressure, exporter-error swallowing, idempotent start/stop, and drain-timeout cancellation).

Reviewer checklist

Apply to every PR that touches an SPI, a hot path, or an SPI consumer:

  • RequestContext threading. Every new SPI method takes ctx: RequestContext as the first arg. Every call to an existing SPI passes ctx (no re-construction from globals).
  • Typed ACL access. Code reads chunk.tenant_id and chunk.acl_labels directly (typed fields), not chunk.metadata["tenant_id"].
  • No raw logging.getLogger. All loggers come from rag_observability.logging.get_logger(__name__). RAG001 pre-commit catches this; if it didn't run, check it's installed.
  • No unbounded queues. Every asyncio.Queue(...) has maxsize=.
  • Pydantic discipline. No Model(...) construction inside a tight loop on trusted data — use Model.model_construct(...) or a dataclass.
  • PolicyEngine call site. Any new code path that reads chunks, ingests data, or emits text consults PolicyEngine / PolicyWriter. The coverage linter (tests/policy/coverage.py) catches misses.
  • Hydration only on survivors. Retrieval code paths use retrieve_ids → rerank → hydrate, not full-Chunk retrieval up front.
  • Batcher for external APIs. New Embedder / Reranker / LLM adapters sit under the Batcher middleware (or document why not).
  • Budgets honored. If the change affects a method with a published p99 budget, the conformance suite still passes.
  • Cache key correctness. New caching code uses the right SPI of the three (Embedding / Retrieval / Answer) for its invalidation rule.
  • BlobRef-safe. Code that consumes Chunk.text handles both str and BlobRef (or hydrates explicitly). (ADR-0007)
  • Plan + budget aware. New retrieval / agent code consumes QueryPlan + RequestContext.budget rather than firing then catching timeouts; expensive nodes carry an estimated_cost and emit StageEvent for the online cost-estimator. (ADR-0008)
  • Index-hint + dtype honored. New VectorIndexBackend code accepts IndexHint at initialize() and respects Embedding.dtype (float32 / int8 / binary) on both write and read paths — no silent up-casting. (ADR-0009)
  • Two-stage rerank shape. New Reranker implementations (Step 2.7+) ship fast_rerank (ChunkRef) + precise_rerank (Chunk) + should_early_exit — not a single-stage rerank. (ADR-0006)

Related