Skip to content

load-testing.md

Latest commit

 

History

History
94 lines (70 loc) · 3.99 KB

load-testing.md

File metadata and controls

94 lines (70 loc) · 3.99 KB
Raw

Guide: load + chaos testing (Step 7.1)

How to validate the GA performance bar — 1000 QPS sustained, p99 < 500 ms, graceful under failure — using the in-process gates (CI) and the Locust suite + chaos harness (cluster).

The two layers

Layer What it proves Where it runs
In-process gates (CI) gateway overhead p99 ≤ 30 ms; graceful degradation under chaos tests/perf/ — deterministic, no infra
Locust suite (cluster) sustained throughput ≥ 1000 RPS; end-to-end p99 < 500 ms a deployed gateway + load generator

The CI gates catch regressions deterministically; the cluster run produces the GA acceptance number (which is hardware/backends-bound and can't be a CI gate — see ADR-0043).

1. Chaos under load (CI-gated, run anywhere)

task chaos-test          # ragctl-free: python -m eval.gateway_chaos_v0.harness --check
# or, targeted:
uv run python -m eval.gateway_chaos_v0.harness --vector-failure-rate 1.0
uv run python -m eval.gateway_chaos_v0.harness --all-fail
uv run python -m eval.gateway_chaos_v0.harness --latency-ms 5

It drives the in-process gateway under concurrent load while injecting backend faults, and asserts graceful degradation: the circuit breaker opens on the failing backend, the fallback ladder keeps serving, and no request 5xx-es. pytest -m perf tests/perf/test_chaos_under_load.py runs it as the gate.

The report (eval/gateway_chaos_v0/report.md) shows, per backend: calls, injected failures, and the resulting breaker state — e.g. 100% vector failure → the vector breaker is open while success stays at 100%.

2. In-process load + latency profile (CI-gated)

task load-test                                        # writes eval/gateway_load_v0/report.*
uv run python -m eval.gateway_load_v0.harness --check  # exit 1 if over the overhead budget
task perf                                              # the p99 ≤ 30 ms overhead gate

This measures the gateway's own per-request overhead (noop backends), the deterministic signal. End-to-end latency under real backends is the Locust job.

3. Locust at scale → the 1000-QPS acceptance (cluster)

Locust is an operator tool, not a project dependency:

uv pip install locust
# Start a gateway (or point --host at your deployment):
uv run uvicorn rag_gateway.app:app --port 8000
# Drive load with the built-in ramp shape:
locust -f eval/gateway_load_v0/locustfile.py --host http://localhost:8000 --headless

The suite runs a weighted mix (/v1/retrieve 5 · /v1/query 4 · /v1/feedback 1 · /v1/status/metrics 1 · /healthz 1) with varied queries (a per-request nonce so the retrieval path is exercised rather than served from cache), and a LoadTestShape that ramps → holds a plateau → ramps down.

Reaching 1000 RPS. One box rarely sustains 1000 RPS through a full pipeline. Scale via env + distributed workers:

LOAD_PEAK_USERS=800 LOAD_HOLD_S=600 \
  locust -f eval/gateway_load_v0/locustfile.py --host https://gw.internal \
         --headless --master                         # + N --worker processes

Env knobs: LOAD_PEAK_USERS, LOAD_SPAWN_RATE, LOAD_RAMP_S, LOAD_HOLD_S.

Acceptance targets

Metric Target Notes
Sustained throughput ≥ 1000 RPS on the plateau, error rate ~0%
End-to-end p99 < 500 ms (dev SLO; ≤ 250 ms SaaS) dominated by backend latency
Error rate under steady load ~0% 5xx ≈ 0
Under chaos (a backend down) no 5xx, breaker opens the chaos gate

Read these off the Locust UI / --csv output on the plateau (ignore ramp-up).

Securing the run

The default gateway is creds-free (anonymous dev tenant). For a secured deployment, add an Authorization header in the Locust user's on_start and point --host at the real gateway.

See reference/perf.md and ADR-0043.